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Overview 


This Logic Model Workshop Toolkit is designed to 
help practitioners learn the overall purpose of a logic 
model, the different elements of a logic model, and 
the appropriate steps for developing and using a logic 
model for program evaluation. This toolkit includes a 
facilitator workbook, a participant workbook, and a 
slide deck. 
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Introduction to workshop 


A logic model is a visual representation of a theory of action or program logic guiding 
the design and implementation of a program or policy and can be used as a tool for build' 
ing a relevant evaluation design. The Regional Educational Laboratory (REL) Northeast 
& Islands administered by Education Development Center created this workshop to help 
groups, such as the research alliances affiliated with the 10 RELs, as well as individual 
alliance members, learn about and build logic models to support program designs and evab 
uations. Based on feedback from alliance members, REL Northeast & Islands learned that 
many of its district- and state-based members would like to build their capacity to develop 
logic models for both evaluating their own programs and working more effectively with 
evaluators whom they engage to conduct evaluations on their behalf. 

This workshop provides a primer on logic modeling and demonstrates how to use logic 
models as a tool for program evaluation. The overarching goals of the workshop are to: 

• Introduce logic models as an effective tool for program or policy design, implemen- 
tation, and evaluation. 

• Practice the elements of a logic model. 

• Provide guidance in appropriate steps for building a logic model for a program or 
initiative. 

• Practice using logic models to develop evaluation questions and indicators of 
success. 

• Provide guidance in how to determine the appropriate evaluation for a specific 
program or policy. 

Versions of this workshop were presented to three REL Northeast & Islands research alli- 
ances in 2013 in two different formats. The Puerto Rico Research Alliance for Dropout 
Prevention participated in a three-hour face-to-face workshop focused on supporting the 
alliance’s effort to generate a common vision for dropout prevention work. The Urban 
School Improvement Alliance and the Northeast Educator Effectiveness Research Alli- 
ance both participated in virtual webinars for a broad audience of practitioners interested 
in developing skills and capacity to develop and use logic models to increase individual 
and group capacity to design and evaluate programs. 

This toolkit provides a complete workbook and slide deck for two sessions. Session I is 120 
minutes long, and Session II is 90 minutes long. The first session focuses on the elements of 
a logic model and the process for developing a logic model for a program or policy. The 
second session provides guidance on how the logic model can be used as a tool to develop 
evaluation questions and indicators of success. While the workshops are designed as 
90-minute to 2-hour sessions, there are many examples of ways to extend the activities and 
the learning when more time is available to go in depth on any particular aspect of the 
materials. These recommended extensions are denoted with a light bulb ( ^ ). The bulk of 
these recommended supplemental activities are best for a face-to-face workshop rather 
than a virtual workshop. However, in many cases guidance is provided on how to approach 
the activity in a virtual context. 


The authors thank the following people from Education Development Center, Inc., for their expertise and 
resources in developing these materials: Katrina Bledsoe, Leslie Goodyear, Brian Lord, and Anne Wang. 
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There are 13 notes in session I and 13 notes in session II that focus on supplemental activ- 
ities and the recommended extensions. 

There are also pre-assignment activities that participants are encouraged to complete 
before the start of the workshop. 

The toolkit includes items needed to successfully present this workshop in a face-to-face or 
virtual setting: 

• Facilitator workbook for sessions I and II. 

• Participant workbook for sessions I and II. 

• Slide deck for sessions I and II (available at http://www.relnei.org/tools-resources. 
html). 

It is recommended that the presenter be familiar with logic models and have some famil- 
iarity with the content that participants might apply to the skills presented in the work- 
shop. Whenever possible, the facilitator should provide examples that are relevant to the 
audience for the workshop (for example, for an audience of school administrators, consider 
examples relevant to their particular work contexts and how they might use a logic model 
in their work). 

Some initial suggestions for tailoring: 

• Organize teams of school-, district-, or state-based leaders to attend together, and 
focus on a particular program or policy. 

• Across contexts, focus on a particular type of program or policy, such as dual 
enrollment (high school and college dual enrollment), educator evaluation, or 
other particular policies or programs relevant to the participants. 

• A pre-assignment is provided at the beginning of the workbook. It is recommend- 
ed that participants have access to the workbook in advance so they can complete 
the pre-assignment in preparation for the workshop. 

• The slide deck that goes with the workbook can be modified to increase relevance 
for the participants. For example, if the group is focused on career and technical 
education, examples could be modified to be more relevant to this content area. 

This material can be challenging for some audiences, and the times suggested are based 
on an experienced facilitator using the toolkit. Therefore, it is important that facilitators 
know their audience. Facilitators new to logic models may want to first practice using the 
workbook and slide deck and become familiar with the purposes of the exercises and deter- 
mine the appropriate pace of the session. 
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Session I. Learning about logic models (2 hours) 


Agenda 


P 

The times here are suggestions, based on 120 minutes for session I. Adjust these times to suit 
your setting, format, and focus. A face-to-face workshop may have some extension activities, as 
described throughout this document, and will therefore be longer than 120 minutes. 




5 minutes 

Introduction and goals 

Facilitator offers introductions followed by an overview of the agenda and purposes of 
the workshop. 

10 minutes 

Introducing the cases 

Facilitator reviews cases that will be used as examples throughout the workshop. 

Activity 1.1 (7 minutes): Participants brainstorm the goals of the programs in the cases 
and the types of questions that might be appropriate to ask about the cases. 

20 minutes 

What is a logic model? 

Facilitator introduces logic models as a useful tool for program design, implementation, 
and evaluation. Facilitator then introduces three types of logic models with three 
different purposes. Facilitator discusses overall idea of inputs-outputs-outcomes that 
drives logic model development and the logic of logic models. 

Activity 1.2 (10 minutes): Participants complete an inputs-outputs-outcomes table for 
the College Ready case. 

65 minutes 

Elements of a logic model 

Facilitator walks through all elements of a logic model. Several individual elements have 
an associated activity. 

Problem statement: Includes discussion of what a problem statement is and 
the questions that need to be considered in developing an appropriate problem 
statement. 

Activity 1.3 (7 minutes): Participants brainstorm their own problem statements. 

• Outcomes: Includes discussion of short- to long-term outcomes and impacts, as 
well as review of what outputs are and how they differ from outcomes. Guidance for 
generating outcomes is provided. 

Activity 1.4 (15 minutes): Participants fill in a table that helps generate outcomes for their 
own examples or for one of the case examples. 

■ Strategies and activities: Focuses on program components and how sequencing and 
clustering of activities should be considered. There is no workshop activity associated 
with this element, but there is an activity in the workbook provided as a suggestion for 
participants to do on their own following the workshop. There is a brief pause here for 
questions. 

Resources: Discusses material and intangible resources. 

Activity 1.5 (7 minutes): Participants brainstorm at least five nonmonetary resources 
available to them in a program. 

■ Assumptions: Discusses examples of assumptions and the importance of being 
explicit about assumptions. 

Activity 1.6 (10 minutes): Participants brainstorm internal and external assumptions for 
their program or initiative. 

10 minutes 

The logic in a logic model 

Facilitator discusses the if-then statements that are embedded within logic models. 
Facilitator then walks participants through the logic of the if-then statements in the 
blended learning case. 

Activity 1.7 (7 minutes): Participants practice with the College Ready case and order a 
series of if-then statements. 

10 minutes 

Next steps 

Facilitator closes with discussion of some next steps for developing a logic model and a 
reminder of how logic models can support program design and evaluation. 

Activity 1.8 (5 minutes): Participants indicate a next step for their work. 
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Pre-assignment 


Consider a program or policy that you lead or are actively involved in designing, imple- 
menting, or evaluating. Come to the session with the following materials or information: 

• For whom is the program or policy designed? 

• What are the main goals or objectives of the program or policy? 

• What is the timeline and duration of the program or policy? 

• What are the major activities or strategies associated with the program or policy? 

Program or policy: 



Main goals 

Timeline and 

Major activities 

For whom 

or objectives 

duration 

or strategies 



For a face-to-face workshop, invite participants to spend a few minutes 
writing notes about a particular program or policy on the worksheet, if 
they have not done so prior to the workshop. Then have them talk to 
another participant about the example they selected. In particular, ask 
them to address the questions: For whom is the program or policy designed 
(and who benefits from the program or policy)? What are the main goals or 
objectives of the program or policy? If there is time, have them also answer 
the question: What are the major activities or strategies associated with 
the program or policy? This pair activity extension will add 15 minutes 
to the overall time for session I. It may be useful to send the participants 
the workbooks before the workshop along with the pre-assignment so that 
they can come to the workshop prepared for this discussion. 
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Goals 


Supporting materials for this section: slide 4- 
Session I of the workshop will: 

• Introduce logic models as an effective tool for program and policy design, imple- 
mentation, and evaluation. 

• Invite participants to practice the elements of a logic model. 

• Provide guidance in appropriate steps for building a logic model for a program or 
initiative. 
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Introducing the cases 


Supporting materials for this section: slide 5. 

This section describes the two cases that are used as the examples throughout the work- 
shop (and provides some additional case examples that may be used if more relevant to the 
particular audience). These examples are based on real programs or policies. These cases 
provide a common language for discussing all aspects of the logic model. Participants may 
also want to draw on their pre-assignment, in which they generated their own example, to 
use throughout the workshop. 

College readiness high school program 

College Ready is a schoohbased college access program for students in grades 9-12. Stu- 
dents are identified for the program based on eligibility for free or reduced-price lunch, 
recommendations from school guidance counselors, and recommendations from grade 8 
English and math teachers. Students participate in monthly meetings as a group with the 
College Ready staff, are provided with one-on-one counseling with College Ready staff, 
are assigned an adult mentor and a peer mentor, and participate in a series of evening and 
summer workshops. In addition, families make a commitment to the program and attend a 
series of workshops specifically designed to prepare the whole family for the college appli- 
cation process. The goal of the program is to significantly increase college attendance 
among low-income students. 

Blended learning schools 

An urban district is going to convert a few of its schools into blended learning schools 
(in which students learn both online and face-to-face with a teacher to personalize their 
instruction). The schools will be using the individual rotation model, which allows students 
within a given course or subject to rotate on an individually customized, fixed schedule 
between online courses and a variety of classroom environments with face-to-face teach- 
ers. Individual students have their own netbook and a unique schedule. The model also 
includes site coordinators and a principal who is involved in daily classroom observation. 
The goal of the model is to improve student achievement and individualize instruction. 
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Additional case examples 


Three additional cases are provided below. Any may be useful for illustrating the key ideas 
in this workshop, and all the cases may be drawn on to explain the concepts embedded in 
a logic model. The cases above were used to support a workshop focused on programs most 
relevant to urban schools, and the first two cases below were used to support a workshop 
for practitioners focused on educator effectiveness initiatives. The final case was used in 
the workshop with the Puerto Rico Research Alliance for Dropout Prevention. In addi- 
tion, if this workshop is being designed for a group of educators focused on a particular 
issue, such as English learner students, consider developing an additional case relevant to 
the participants. 

Redesigning a district’s educator evaluation process 

A school district wants to review and update the teacher evaluation process that it has 
used for more than 10 years. The new system must reflect the new state guidelines for eval- 
uation, which include a requirement for multiple measures, including a student-learning 
measure. However, much is left to the district to determine: How decisions will be made, 
what measures will be used, who will conduct the evaluations, and how the evaluation 
process will be managed and supported. The district has determined, in keeping with state 
guidelines, that the new evaluation will assess teachers’ professional practice and their 
impact on student learning. The district leadership would like the system to be supported 
by teachers, to effectively differentiate among teachers, to support teachers’ ongoing pro- 
fessional growth, to lead to improvements in teacher practice, and ultimately to positively 
influence student learning. 

Professional development initiative for science teachers 

A state department of education is implementing a new professional development program 
for science teachers in secondary schools that focuses on promoting inquiry-based 
approaches. The state will partner with informal science partners, such as museums, and 
with industry and will develop a series of content courses for teachers. The professional 
development will use research-based materials and practices so that teachers can build a 
deep foundation of knowledge that will allow them to grow professionally. In addition, 
districts will create professional learning communities and ongoing science instruction 
development opportunities to support inquiry approaches. There will also be profession- 
al development offerings that focus on leadership development to support coaches, lead 
teachers and principals, and curriculum directors in science learning. The goal of this 
program is to enhance teachers’ knowledge in science, increase student participation in 
science courses, and increase student learning and achievement. 

Universal prekindergarten policy 

Massachusetts has implemented a policy to provide prekindergarten education to all chil- 
dren through a blend of private and public providers. According to Early Education for All, 
70 percent of three- and four-year-olds in Massachusetts attend a formal early education 
and care program. The challenge is to ensure that the programs they attend are high- 
quality ones, because as decades of research document, only high-quality prekindergarten 
delivers lasting benefits, particularly for low-income children. 
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Activity 1.1: Discussion of cases 

• Individually. 

Working on your own for about two minutes and using the worksheet below, consider the 
College Ready or blended learning case above. What are the goals of the program? What might 
we want to know about it? Consider questions of implementation, effectiveness, and impact. 

• In large-group discussion. 

What are your ideas about the goals of the program and what do you want to know about it? 


What are the goals of What do we want to know about 

the program or policy? the program or policy? 



For a face-to-face workshop, consider adding a teamwork component to 
activity 1.1, in which the participants discuss one of the cases above in 
small groups. Consider allowing people to select their own case to focus on 
or divide them into groups to discuss one or more cases. Then ask partici- 
pants to briefly share the goals and some of their questions. This discussion 
will add 15 minutes to the overall time for session I. 
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What is a logic model? 


Supporting materials for this section: slides 6-9. 

This section provides a primer on logic models, including different types of logic models 
and their potential purposes. 

In the most basic terms, logic models provide a kind of map for a program or initiative, 
helping clarify a program or policy’s destination, the pathways toward the destination, and 
markers along the way. 

Consider: 

• Where are you going? 

• How will you get there? 

• What will tell you that you have arrived? 

Logic models provide a simplified picture of the relationships between the program or 
policy inputs (resources, strategies, activities) and the desired outcomes of the program. 

Logic models present a theory of action or change that drives the program or policy and 
makes explicit any assumptions about both the resources at the disposal of the program 
and the rationale behind the effort. 

A logic model is valuable in supporting: 

• Program planning. 

• Program implementation. 

• Program monitoring. 

• Program evaluation. 

Why use a logic model? A logic model: 

• Brings detail to broad goals. 

• Helps identify gaps in program logic and clarify assumptions. 

• Builds understanding and promotes consensus. 

• Makes explicit underlying beliefs. 

• Helps clarify what is appropriate to evaluate and when. 

• Summarizes complex programs for effective communication. 

A logic model is useful in designing program and policy evaluation, because a logic model 
helps clarify both what the program, initiative, or policy is and what it is not. This kind of 
clarification is helpful in building an evaluation design that can capture the program’s or 
policy’s influence and impact. 

What are the limitations of a logic model? A logic model is not: 

• A strategic or fully developed plan for designing or managing a program or policy. 

• An evaluation design or evaluation method. 

While logic models are useful tools for building program plans or evaluation designs, addi- 
tional work is necessary to create both programmatic and evaluation plans. 
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Types of logic models 


Not all logic models are the same, nor are they designed for the same purpose. Just as logic 
models may aid in program design, implementation, and evaluation, the type of model 
developed varies somewhat based on its purpose. There are three main types of logic 
models: 

• Theory approach. 

• Activities approach. 

• Outcomes approach. 

Theory approach models 

Logic models that describe the overall theory of change provide a “big picture” of the 
program and may be useful for program design and overall communication of the program 
theory. These models provide a clear description of why the developers believe the program 
or policy will be effective in achieving the goals. In the blended learning case, a theory 
approach logic model might help clarify the assumptions implicit in the push for alter- 
natives to traditional brick and mortar classrooms and describe the relationship between 
initiating blended learning and the expected outcomes for students who participate. 

Consider: 

• What might be the logic in a theory approach model for your own program or 
policy? (The “big picture” theory of change about your initiative?) 

Activities approach models 

Activities approach models focus on laying out the specific strategies and activities associ- 
ated with a program. These models closely examine the relationship among the activities, 
considering sequence and timing of implementation, as well as how the activities link to 
outcomes. This type of logic model is most useful in program implementation, monitor- 
ing, and management. In the College Ready case, this type of logic model would consid- 
er the elements of the program and how they would be optimally ordered and managed. 
For example, what role would the different mentoring components have? How would they 
relate to one another? In this type of model, relationships among variables are made explic- 
it with arrows, concentric circles, and other graphic representations of relationships. 

Consider: 

• Why consider the sequence and relationship among activities in a logic model? 
How might that help you? 

Outcomes approach models 

Outcomes approach models are most useful for program evaluation. They consider the 
strategies and activities as they relate to the desired results of a program or policy. In these 
models, the focus is on outcomes, and they often divide the outcomes into short-term out- 
comes, long-term outcomes, and impacts. A theory of change drives these models just as it 
does the others. But in an outcomes approach logic model, the emphasis is on examining 
the outcomes and making the case that the program or policy is responsible for the desired 
outcomes. 
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Consider: 

• Why divide outcomes into short term and long term? What is the difference 
between outcomes and impacts? 



For a face-to-face or small virtual workshop that has the option for par- 
ticipants to discuss, the “consider” questions could be posed for discussion 
among the whole group or among small groups. This will add 10 minutes 
to the overall time for session I. If less time is available, these questions can 
be run through quickly or referenced without discussion, as they are not 
essential to understanding a basic logic model. 


Inputs-outputs-outcomes 

Supporting materials for this section: slides 10-14- 

In its simplest form, a logic model is a graphic representation of the relationship among a 
program’s or policy’s inputs (what is invested in the program), the outputs 1 (what is done 
with these investments), and what the outcomes are (what are the results). 

Take a simple example: You have a headache and you want it to go away. 

• What is the input? 

° Quiet time. 

° Water. 

0 A hot compress. 

° Two aspirin. 

• What is the output (that is, what do you do with the inputs)? 

0 Sit quietly for five minutes. 

° Drink a full glass of water. 

° Put hot compress on. 
o Take aspirin. 

• What is the outcome? 

° You are more relaxed. 

0 You are hydrated. 

° Your headache goes away. 

° You are able to return to your work. 


1. The use of the term “output” here differs from its use when outputs are included as a specific element 
in a complete logic model. The use here is in terms of the general inputs-outputs-outcomes logic of 
a logic model. In this case, outputs do not refer to the measurable, targeted activities that are actually 
completed; rather, they refer more generally to what is done with the inputs to achieve the out- 
comes — that is, as one part of the overall logic embedded in a logic model. While possibly confusing, 
the main idea here is that a logic model sets up a logical progression from resources to activities to 
outcomes. The section that discusses the specific elements of a full logic model separates strategies 
and activities from outputs. 
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Below is an example from the blended learning case. Some base inputs, outputs, and out- 
comes have been laid out, reflecting the overall logic of the program. 


1 Inputs 

Outputs 

Outcomes i 

Existing technology 
infrastructure 

Technology integration staff 
person for three schools 
Teachers’ enthusiasm in three 
schools 

Technology integration grant 

Infrastructure audit completed 

• Completion of six days of 
summer teacher professional 
development 

• Six blended learning 
classrooms established 

Teachers’ reported use of 
diverse instruction strategies 
increases 

• Student engagement increases 

• Student achievement on 
district-wide assessments 
improves 



To extend the activity in a face-to-face workshop, after presenting the 
headache example, ask participants to think of another example. Or 
suggest the example of the problem being a house on fire. Then ask them 
to consider the inputs, outputs, and outcomes associated with solving that 
problem. Allow them time to generate some ideas on a piece of paper and 
then ask them to provide some examples. This will add 10 minutes to the 
overall time for session I. Here are some possible answers: 


• Inputs: 

° Ladder. 

0 Hose with water. 

° Firefighters. 

• Outputs: 

o Firefighters climb the ladder. 

° Firefighters spray water on the fire. 

° Firefighters bring the people down the ladder. 

• Outcomes: 

° The fire stops. 

° The people are saved from the fire. 
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Activity 1.2: Inputs-outputs-outcomes 

Using one of the example cases (College Ready), consider the inputs, outputs, and outcomes. 
This activity helps illustrate the basic purpose and premise of logic models, but what is gen- 
erated is not an actual logic model. Rather, this activity is designed to help you understand 
the overall logic undergirding logic models. For more information on the elements of the logic 
model, see the next section. 


Inputs 


Outputs 


Outcomes 



For a virtual workshop, if the software functionality allows, consider 
extending activity 1.2, and ask participants to select from a series of multi- 
pie-choice options reflecting what the inputs, outputs, and outcomes might 
be for the College Ready case. Use the same set of options for the inputs, 
outputs, and outcomes, and have participants choose what they see as 
inputs, outputs, and outcomes (participants can choose as many as apply). 
Spending time on the examples and doing this activity will add 10-15 
minutes to the workshop. Here are the options for the activity: 


• Parent involvement increases. 

• Staff. 

• College applications increase. 

• School space and resources. 

• Course for parents. 

• College attendance increases. 

• Volunteer mentors. 

• Mentoring students. 

• College acceptance increases. 

• Guidance counselor meetings. 

• Teacher time. 

• Student meetings. 
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Elements of a logic model 


Supporting materials for this section: slides 15-18. 

This section provides an overview of — and opportunities to practice — all elements of a 
logic model, which are listed below. The pages that follow delineate these elements and 
use examples from the two cases. The elements are discussed in a different order from their 
presentation in the simple logic model on slide 15 (and in appendix A) because the order 
below makes the most logical sense when working through the different elements of a logic 
model. For example, once a problem statement is determined, thinking about the overall 
goals would be the next logical step, so discussing outcomes should be considered next. 

• Problem statement. 

• Short- and long-term outcomes. 

• Impacts. 

• Outputs. 

• Strategies and activities. 

• Resources (inputs). 

• Assumptions. 

Problem statement 

The problem statement is the problem or challenge that the program or policy is designed 
to address. 

Consider: 

• What is the problem or issue? 

• Why is this a problem? (What causes the problem?) 

• For whom (individual, household, group, community, society in general) does this 
problem exist? 

• Who has a stake in the problem? (Who cares whether it is resolved?) 

• What is known about the problem, issue, or people who are involved? What 
research or experience is available? What do existing research and experience say? 

Finally, ask whether the problem statement is too big or too small. The final problem state- 
ment should be targeted and specific, but it should not be a simple restatement of the 
program as a need. For example, in the blended learning case, “Students lack access to 
their own netbook” is really a statement about the lack of the program. The problem state- 
ment should address the real issues underlying the need for the program, such as “Students 
have limited one-on-one attention from teachers.” 
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Activity 1.3: Problem statement 

Consider the problem statement most appropriate to the challenge you face in your work, 
related to a program you have in place or one you would like to initiate. In other words, consider 
the problem for which your program or policy is the “answer.” Brainstorm key ideas associated 
with the relevant problem. Model your brainstorm after the example below, in terms of brevity. 
There are several different ideas reflected below; in your brainstorm, consider all potential 
ideas or elements of the problem. Remember that a problem statement should be targeted 
and specific. 

Case: Blended learning 

• Students are not actively engaged in their learning. 

• Courses are sometimes monotonous. 

• Students have limited one-on-one attention from adults. 

• Students’ courses are not personalized. 

• Students are all expected to work at the same pace. 

Your brainstorm: 



For a virtual workshop, if the software functionality allows for a chat func- 
tion, ask participants to type in some bulleted examples from their own 
contexts. As facilitator, comment on their examples and guide them to 
specific and focused problem statements. Remind participants that the idea 
is to be sure that the problem statement they identify is targeted and specif- 
ic; it should not read as a statement of the need for the program specifically. 


Outcomes 

Supporting materials for this section: slides 20-27. 

While outcomes are not the next item one sees when one looks from left to right across 
a traditional logic model, they are a logical next step to discuss when examining the ele- 
ments of a logic model. Outcomes should be thought of next because knowing the goal in 
relation to solving the problem defined in the problem statement will help in developing a 
plan for achieving the outcomes. After identifying the problem, it makes sense to identify 
the overall goal, or outcomes, next. 
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Outcomes ask, “What difference does it make?” In other words, what is the difference that 
the resources, and strategies and activities, taken together, should have on the various par- 
ticipants in these efforts? For example, in the College Ready case, “What difference does 
the mentoring program have on students’ daily school attendance?” or “What difference 
does the mentoring program have on students’ grade point average or college attendance?” 

Outcomes usually come in stages and fall along a continuum from short- to long-term out- 
comes. The language to describe these outcomes varies; this workbook refers to short-term 
outcomes, long-term outcomes, and impact. Other terms include: 

• Short-term: initial, immediate, proximal. 

• Long-term: medium-term, intermediate, midpoint. 

• Impact: long-term, final, ultimate, distal outcome. 

Short-term outcomes 

Short-term outcomes are the most immediate and measurable results for participants 
that can be attributed to strategies and activities. For example, a program that promotes 
increased parent engagement in students’ college planning might have a short-term goal of 
increased parent participation in the provided parent sessions. 

Long-term outcomes 

Long-term outcomes are the more distant, though anticipated, results of participation in 
strategies and activities. When it comes to short- and long-term outcomes, it is good to 
think about the overall timeframe for the program. Sometimes, short term is considered to 
be as short as six months or as long as three years. Long term might be two years or as long 
as six years. The important point is to consider the program and identify the timeframe, 
specific to the initiative, for short- and long-term outcomes. For example, a long-term 
outcome for a program focused on increasing college readiness may be improved academic 
performance of participants in the program. 

Impacts 

Impacts are the desired outcomes of long-term implementation of strategies and activities 
that depend on conditions beyond the program’s scope of strategies. These may be called 
the “blue skies” or the big picture types of objectives that are more distant from the actual 
strategies and activities and less within the control of the program or policy to realize. 
Often these are considered to be 7-10 years after initial implementation. For example, an 
impact of a college readiness program might be an increased percentage of students grad- 
uating from post-secondary institutions after participating in their high school’s college 
readiness program. 

Outputs versus outcomes 

Some logic models include both outputs and outcomes. Outputs differ from outcomes in 
that they capture data about what is done rather than what is expected to be achieved 
as a result of what is done. Outputs can best be described as activity data and are useful 
for tracking program implementation. They often provide detail about the breadth and 
reach of strategies and activities. Outputs capture size and scope; they describe or count 
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strategies and activities, such as the number of parent sessions delivered, the program par- 
ticipation rates, the number of materials developed or distributed, and so forth. Using the 
College Ready program as an example, another way to think about the difference between 
outputs and outcomes is to consider the questions: 

“Is the parent education program being delivered as intended?” (output question) 
versus 

“Are parents who participated in the parent education program becoming more actively 
involved in their children’s education?” Or, a long-term outcome might be, “Is the college 
acceptance rate for participating students increasing?” (outcome question) 

It is important not to confuse outputs for outcomes. A program that is good at delivering 
activities and services may achieve its outputs without achieving its outcomes. Yet, it is the 
outcomes that make the difference in response to the problem identified. 


Activity 1.4: Focus on outcomes 

Being clear about program outcomes is essential for both focused program implementation 
and effective evaluation. The table below is designed to promote a step-by-step approach to 
outcome development. The columns are: 

• Who is the target? Who is the group targeted with the strategy? Is it students? Parents? A 
school? In this example, based on the College Ready case, the target is participating high 
school seniors in three high schools that participate in the program. 

• What is the desired change? Use an action verb to demonstrate a kind of change or an 
impact. For example: increase, improve, engage. 

• In what? What is the activity, strategy, or program in which the target population is going to 
enact this desired change? What is the resulting action in which the target population will 
engage to achieve the goal? 

• By when? Here is where the timeline for outcomes is clarified. Is a particular outcome a 
short- or long-term outcome? 

Enter an example from your own context, related to a program or policy initiative you have 
in place or would like to develop. If you prefer, consider the College Ready or blended learning 
case and fill in the table with a relevant example from one of those cases. 


What is 
the target? 

What is the desired 
change? (action verb) 

In what? 
(results) 

By when? 

Participating high school 

Increase 

Completed and 

By June 2014 

seniors in three urban 


submitted applications 


high schools 


to post-secondary 




institutions 
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For a virtual workshop, if the software functionality allows, ask participants 
to enter their responses to the table above in the chat pod. For a face-to- 
face workshop, have participants turn to their colleagues and explain their 
example, in groups of two or three, or invite a participant to read off one of 
his or her examples and discuss it in a large group. 


People occasionally do not stick to the columns and instead present a sentence for 
their outcome. It is important to follow through with this chart and fill out each 
column. Doing so ensures that each outcome has a clear target, action, and timeframe. 
Encourage participants to follow this format, and if someone presents an example that 
is not in the format, work as a group to place the example into the table columns. This 
activity will add 10 minutes to the workshop. 


Outcomes checklist 

Consider the following criteria when examining outcomes: 

• Are the outcomes important? Do they represent significant changes or improve' 
ments that are valued by participants and key stakeholders? Outcomes may be 
achievable but not really worth the effort. If the outcomes were achieved, would 
anyone care? 

• Are the outcomes reasonable? Are the outcomes connected to one another and 
linked in a reasonable order (from short term to long term to impact)? Is it likely 
that one will lead to the next? Another way to think about this is to consider 
the if-then statements (or logic statements) embedded in a chain of outcomes. 
For example, using the College Ready case, will increased parent participation in 
workshops on college readiness lead to students’ completing more college appli- 
cations? Will access to online courses lead to increased student engagement 
and achievement? Sequence and timing of activities and intended outcomes are 
important to consider. 

• Are the outcomes realistic given the nature of the problem and available resourc- 
es and abilities? Will the program lead to or help contribute to these outcomes? 
(Be careful to ensure that the outcomes are realistic given the level of effort.) In 
other words, is it realistic to expect an increase in student achievement from one 
parent education class? Ask hard questions about the outcomes as they relate to 
the actual program or policy. 

• Are unintentional or possibly negative outcomes being attended to? It is important 
to anticipate and consider the unintended or potentially negative outcomes that 
might result from the set of strategies and activities. What are potential negative 
effects of the program or policy? What else might happen that is different from 
what is intended? How else might the sequence of events unfold? For example, 
could access to online courses lead to lower student attendance? Considering the 
unintended consequences allows program and policy designers to consider how to 
prepare for these possible outcomes and also helps evaluators be attuned to these 
possible consequences in the evaluation design. 
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Another common set of criteria for outcomes is the S.M.A.R.T. goals. These are: 

• Specific. 

• Measurable. 

• Action oriented. 

• Realistic. 

• Timed. 

Strategies and activities 

Supporting materials for this section: slides 28-29. 

Strategies and activities are the program components, or the game plan for the program 
or policy. This is an inventory of all the strategies and activities designed to achieve the 
outcomes. However, it is more than a simple listing of activities. There are two questions 
to ask when inventorying the activities, services, products, and events that make up the 
program or policy: 

• What is the appropriate sequence or order of these activities? 

Consider the College Ready case: It may be important that the mentoring element 
of the program come prior to the delivery of the parent workshop series. Or 
perhaps these activities should be concurrent. Consider the appropriate order of 
activities and how they relate to one another. 

• Are there certain activities that, taken together, add up to a kind of overall strate- 
gy? Do certain activities bundle or cluster together? 

Consider the blended learning case: Perhaps there is a series of training needs 
related to instituting the blended learning model, such as new professional deveh 
opment offerings for teachers, new demands on the technical support staff at the 
schools, and new requirements for paraprofessional support to the classrooms, 
that bundle together as an overarching strategy. Perhaps this is the professional 
training strategy. This may be different from other strategies associated with the 
initiative, such as infrastructure or family engagement. Creating these clusters of 
activities helps streamline the logic model and supports evaluation; the evaluation 
will then assess a set of strategies, rather than individual activities. 

Here is a brief example from the blended learning case, illustrating both the sequence in 
which activities might unfold and how specific activities relate to a set of core strategies. 


1 Activities 

Sequence 

Strategy | 

Develop teacher training materials 

1st 

Professional training 

Deliver summer institute for teachers 

2nd 

Professional training 

Conduct technology audit 

lst a 

Infrastructure 


a. Occurs concurrently with the develop teacher training materials activity and prior to the deliver summer 
institute for teachers activity. 
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The activity below, in which participants generate and organize their strat- 
egies and activities, will add considerable time to the workshop, but in a 
longer face-to-face workshop in which participants are prepared to work 
on a particular program or policy, this is the time to conduct this activity. 
Otherwise, participants may want to address this on their own after the 
workshop. 


It may be premature to ask participants to fill out this type of chart during the work- 
shop. However, if the participants are focused on a particular issue or are working 
together to build a logic model, a discussion of the key strategies can be useful in 
setting the stage for further work inventorying the range of activities related to the 
program or policy. For example, in the workshop with the Puerto Rico Research Alli- 
ance for Dropout Prevention, participants brainstormed a list of activities related to 
dropout prevention and then tried to organize the activities into a set of core strat- 
egies. If the participants are working collectively on an issue and may be building a 
logic model together, an initial conversation about the core strategies could be very 
valuable at this stage of the workshop. 


If you choose to do this activity, here is a possible format: 

• Either working in pairs or individually, participants brainstorm activities that 
they can write on sticky notes. Allow them 10-15 minutes to generate activities 
related to their program. 

• Participants then share one activity from the sticky notes with the whole group. 

• Ask participants with similar activities on their sticky notes to come up and 
place these sticky notes next to the others on the wall. 

• Go around the room until all sticky notes have been placed on the wall in 
groups of related activities. 

• Once all sticky notes are grouped in related activities, ask participants to come 
up with a name that describes the related activities and write it next to them. 
The name they come up with will be the core strategy that the grouped activi- 
ties represent. 


If you include this activity, assume another 30 minutes to brainstorm activities and 
begin to generate the core strategies. This may also be a useful extension activity to 
have participants conduct after the workshop, rather than something to do during this 
workshop. However, you might explain this activity and suggest to the participants 
that they employ this type of process to generate their core strategies. 
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Additional activity: Strategies and activities in sequence 

Consider a series of activities that are a part of your own work. List some relevant activities, 
give the sequence or order in which they are supposed to occur, and consider the overarching 
strategy within which these activities fall. In other words, does your chosen program or ini- 
tiative have a core set of strategies that guide the activities, events, programs, and the like 
that you provide? This activity is suggested as something to work on independently after the 
workshop. 


Activities Sequence Strategy 



It may be wise to pause and take questions at this point in the workshop. 
Slide 30 reflects that pause in the workshop. 


Resources (inputs) 

Supporting materials for this section: slides 31-33. 

Resources include both the material and the intangible contributions that are or could 
reasonably be expected to be available to address the problem. 

• Material resources include: 
o Money. 

° Materials and equipment. 

• Intangible resources include: 

° People. 

° Time. 

° Partnerships. 
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Activity 1.5: Intangible resources 

Brainstorm intangible resources (example from College Ready case). 

• Community mentors. 

• Local university space for parent meetings. 

• Volunteer college admissions directors for application workshop. 

• Student volunteers for childcare at parent meetings. 

What intangible resources are at your disposal? Brainstorm at least five nonmonetary 
resources that are available to you in a program you operate or manage: 


Assumptions 

Supporting materials for this section: slides 34-36. 

Assumptions are beliefs about participants, staff, and the program, as well as about how 
change or improvement may be realized. Being explicit about assumptions is one of the first 
and most important considerations during program design, implementation, and evaluation. 

Consider the College Ready case: The program assumes that students who participate want 
to go to college and that college enrollment will lead to a better life for participants. Often 
the assumptions embedded in a program or policy are critical to the success or failure of the 
overall initiative. Assumptions may be internal (assumptions about participants, resources, 
and how the program will function) or external (beliefs about how change occurs, values 
embedded in the program, or findings from prior research). 

Recall the headache example from earlier in the workshop. You had a headache, you tried 
a few things to get rid of it (water, aspirin), and you felt better. The outcome was that the 
headache went away. However, between the problem (the headache) and the outcome (no 
headache) were several assumptions. For example, you assumed that you did not have an 
allergy to aspirin, that there was no loud noise persisting in the background, and so forth. 
Clarifying and making explicit the assumptions behind the program, in terms of both the 
specific elements related to implementation and the assumptions embedded in the theory 
driving the initiative, are critical to developing a thoughtful logic model. 

In the blended learning case, internal assumptions might include a belief that the school 
leadership will support the blended learning classrooms going forward and that the staffing 
available will be adequate to support implementation. External assumptions might include 
a belief that access to varied learning modalities will increase student engagement and 
that increased student engagement will yield increased student achievement. These exten 
nal assumptions are both related to the theory of action or change driving the initiative. 
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Activity 1.6: Uncovering internal and external assumptions 

Consider your program or initiative. Brainstorm the range of assumptions embedded in the 
program design and in the overall theory of action driving the initiative. 


Internal assumptions External assumptions 



For a virtual workshop, if the software functionality allows, create two sep- 
arate chat pods, one for internal assumptions and the other for external 
assumptions. Have participants enter in the chat pods their own internal and 
external assumptions. Then comment on the responses as they are typed in. 


For a face-to-face workshop where the group includes representatives of several organi- 
zations and is not a group that works together, have the participants do some person- 
al brainstorming and then split into smaller groups to share their ideas. If the group 
shares a common content area or is working on building a logic model together, have 
them begin to uncover the assumptions that each participant carries about potential 
program beneficiaries, about the resources available for the program, about the poten- 
tial impact of the program, and about the way that change may occur. Ask participants 
to consider their own beliefs and values as they relate both to the problem and to the 
program intended as a response to the problem. As facilitator, if the group is small 
enough and the focus is on a particular program or initiative, facilitate one group con- 
versation with a note-taker recording the comments on the board or flip chart paper. 
Brainstorming and sharing in small groups will add 15 minutes to the overall time for 
session I; brainstorming assumptions related to a specific program will add 25 minutes. 
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The logic in a logic model 


Supporting materials for this section: slides 37-39. 

The purpose of this section is to understand the logic embedded in a logic model and 
recognize the need to identify the possible relationships and connections among various 
elements of a logic model. 

Understanding these if-then relationships is essential to uncovering the theory of action 
or theory of change driving a program or policy. Access to and application of resourc' 
es will lead to programs that reach the target participants, and when these populations 
are reached by such programs, unmet needs will be met and circumstances will change, 
solving the problem that initiated this work. 

Consider the blended learning case: 

• If the district applies funds to support blended learning in three schools, then the 
schools will provide teachers with professional learning opportunities and estab- 
lish the infrastructure to support blended learning. 

• If the schools provide teachers with professional learning opportunities and estab- 
lish infrastructure to support blended learning, then students’ access to online 
courses and varied learning environments will increase. 

• If students have increased access to online courses and varied learning environ- 
ments, then teachers will be able to personalize instruction, and the students will 
be more engaged in their learning. 

• If teachers personalize instruction and students are more engaged in their learn- 
ing, then students will be able to master content and develop their skills at a pace 
appropriate to the individual student. 

• If students master content and develop their skills at a pace appropriate to the 
individual student, then they will perform better on standardized assessments of 
their learning. 

The then clause in one statement becomes the if clause in the next statement. This is 
important; when the language changes from then to if, the intention of the statement may 
change. In some logic models, if-then statements are written right into the model to make 
the theory of change explicit. 
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Activity 1.7: If-then statements 

Consider the College Ready case. Move the statements around to make a series of logical 
if-then statements below. Consider the sequencing of events. The statements below include 
strategies and activities, traditional outputs, and outcomes. 

IF THEN/IF 

THEN/IF THEN/IF 

THEN/IF THEN 

1. We develop a series of college readiness workshops for parents. 

2. Parents help their students with the application process. 

3. We recruit parents to participate in the workshops. 

4. Parents better understand the timelines and demands of the college application process. 

5. Students meet financial aid and college application deadlines. 

6. Parents attend the workshops. 



For a virtual workshop, if the software functionality allows, present the 
participants the if-then statements above and have them select the first, 
second, third, and other statements. Then move the statements around on 
the screen to show the order of the if-then statements. This virtual activi- 
ty will add 5-7 minutes to the overall time for session I. 


For a face-to-face workshop, hand out all the statements in an envelope and have a 
group work together to order the statements and paste the statements in order onto a 
big sheet of paper. They can then present their order to the whole group. For a face-to- 
face workshop, this activity will add 15 minutes to the overall time for session I. 
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Next steps 


Supporting materials for this section: slides 40-42. 

In building a logic model, it is important to consider the following questions: 

• Do I understand the different elements of the logic model and how they differ? 

• Who should I consult in developing the logic model? What colleagues and stake- 
holders should be participants in developing the logic model? 

• Who will be responsible for seeing this through? 

• How do I know I have captured the theory of action guiding the program? 

• How will we use the logic model once it is developed? 


Activity 1.8: Next steps 

Consider what your next step might be with regard to logic models. Consider where you are in 
the development of a new program or in an evaluation of a program already in place. How can 
logic models support this work? 



For a virtual workshop, if the software functionality allows, ask partici- 
pants to type in a next step in the chat pod. For a face-to-face workshop or 
a virtual workshop with a small group and audio capability, have all partic- 
ipants state what they have in mind as a next step in a few words. 
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Final thoughts 


Supporting materials for this section: slides 43-44. 

Here are a few quick reminders about what a logic model is and what it is not. 

A logic model is: 

• A graphic representation of the theory of change driving a program or policy. 

• A framework for planning, implementation, monitoring, and evaluation. 

A logic model is not: 

• A strategic or fully developed plan for designing or managing a program or policy. 

• An evaluation design or evaluation method. 

While a logic model is not a strategic or fully developed plan or an evaluation design or 
evaluation method, it can be useful in developing any of these more detailed resources. 

A logic model is likely to be much more effective, useful, and honest if the process of 
generating the logic model has engaged a broad range of stakeholders during the design 
process. Including key voices such as staff, parents, students, funders, and others in discus- 
sions about program design and evaluation will promote the buy-in and ongoing support of 
these participants as well as increase the authenticity of the model. 

Logic models should be living documents that are referred to throughout the life of the 
program and the evaluation and should be amended as needed. They are also helpful to 
guide a program as it evolves and to ensure that the work of the program remains focused 
on the key goals and outcomes. 

Logic models are useful for program evaluation, especially when evaluation is considered 
in concert with creating the logic model at the early stages of program development. It is 
much better to consider evaluation at the outset of development of a program or policy 
rather than after or halfway through program implementation. 

Good luck with this work, and please contact us with questions! 
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Session II. From logic models to program and policy evaluation (1.5 hours) 



Agenda 

5 minutes 

introduction and goals 

Facilitator reviews the goals of the session and the agenda. 

8 minutes 

Review of logic models 

Facilitator reviews what was learned about logic models, what they are useful for, and 
what limitations they have. A graphic of a simple logic model and a logic model from one 
of the cases will be reviewed. If participants come with their own draft logic models, two 
will be selected for presentation and discussion. 

15 minutes 

Introducing evaluation 

Facilitator introduces the types of questions that evaluation is designed to answer, the 
value of implementing evaluation at the onset of program development, and the role that 
logic models play in supporting evaluation. Two purposes of evaluation are presented: 
evaluation that focuses on improvements and evaluation that focuses on proving or 
demonstrating the impact or outcomes of an initiative. 

Activity II. 1 (7 minutes): Participants brainstorm ways they know their program or policy 
is achieving its goal. 

13 minutes 

Moving from logic model to 
evaluation questions 

Facilitator begins with more about types of evaluation questions, followed by guidelines 
for good questions. Facilitator then introduces the idea of different audiences desiring 
different information about a program or policy and therefore asking different questions. 
Participants will be introduced to a table that delineates different types of audiences, 
questions, and uses of evaluation. 

Activity 11.2 (4 minutes): Participants brainstorm formative or summative evaluation 
questions about their own program or policy or for one of the sample cases. 

Activity 11.3 (4 minutes): Participants practice generating questions for different audiences. 

25 minutes 

Generating indicators 

Facilitator introduces the concept of indicators and provides an overview of how 
indicators are generated from the logic model, specifically from the strategies and 
activities and outcomes sections of the model. Facilitator provides an example of this for 
the College Ready case. Section closes with a discussion of qualitative and quantitative 
indicators and the use and value of both types of measures in an evaluation. 

Activity 11.4 (3 minutes): Participants brainstorm indicators of the flu. 

Activity 11.5 (3 minutes): Participants brainstorm outcome and process indicators based 
on one of the sample cases. 

15 minutes 

Building an evaluation 
design 

Facilitator begins this section with the question, “What type of evaluation is right for 
you?" and suggests that determining the purpose — formative, summative, or hybrid — is 
critical to building an evaluation. Facilitator transitions to more discussion about data 
collection, specifically considering the types of data available to participants. Types of 
data, both quantitative and qualitative, are reviewed. Then facilitator introduces the 
data collection framework tool, which outlines the outcomes of interest, data sources, 
responsible parties, and timeline. This is followed by discussion of evaluation design, as 
distinct from the data collection framework. 

Activity 11.6 (5 minutes): Participants brainstorm their own data sources. 

6 minutes 

Putting it all together 

Facilitator discusses an evaluation prospectus or overview and the key questions to 
consider when generating this short document, which can serve as the “calling card” 
for an evaluation, either for potential funders or for potential evaluators. The facilitator 
closes with a presentation of a Gantt chart as a useful tool for managing an evaluation 
and considering realistic timelines and deliverables. 

3 minutes 

Review 

Facilitator closes workshop with thank you and invitation to be in touch with further 
questions. 
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Pre-assignment 


Based on the work in session I, participants may come to the workshop with a draft logic 
model for a particular program or policy. If participants do not have their own draft logic 
model, they should familiarize themselves with the sample logic models in appendixes C 
and D, as these will be drawn on for examples throughout the workshop. 

A sample logic model template is provided in appendix B and may be used to generate a 
simple logic model. Participants will use this logic model to guide their work in the session. 
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Goals 


Supporting materials for this section: slide 48. 

The purpose of session II is to demonstrate how logic models may be used as a tool specific 
to developing a program or policy evaluation. The session will: 

• Reintroduce logic models as an effective tool, specifically for evaluation. 

• Invite participants to practice using logic models to develop evaluation questions 
and indicators of success. 

• Provide guidance in how to determine the appropriate evaluation for a specific 
program or policy. 
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Example cases revisited 


College readiness high school program 

College Ready is a school-based college access program for students in grades 9-12. Stu- 
dents are identified for the program based on eligibility for free or reduced-price lunch, 
recommendations from school guidance counselors, and recommendations from grade 8 
English and math teachers. Students participate in monthly meetings as a group with the 
College Ready staff, are provided with one-on-one counseling with College Ready staff, 
are assigned an adult mentor and a peer mentor, and participate in a series of evening and 
summer workshops. In addition, families make a commitment to the program and attend a 
series of workshops specifically designed to prepare the whole family for the college appli- 
cation process. The goal of the program is to significantly increase college attendance 
among low-income students. 

Blended learning schools 

An urban district is going to convert a few of its schools into blended learning schools 
(in which students learn both online and face-to-face with a teacher to personalize their 
instruction). The schools will be using the individual rotation model, which allows students 
within a given course or subject to rotate on an individually customized, fixed schedule 
between online courses and a variety of classroom environments with face-to-face teach- 
ers. Individual students have their own netbook and a unique schedule. The model also 
includes site coordinators and a principal who is involved in daily classroom observation. 
The goal of the model is to improve student achievement and individualize instruction. 
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Additional case examples 


As with the first session, three additional cases are provided below. Any may serve as useful 
tools for illustrating the key ideas in this workshop. 

Redesigning a district’s educator evaluation process 

A school district wants to review and update the teacher evaluation process that it has 
used for more than 10 years. The new system must reflect the new state guidelines for evah 
uation, which include a requirement for multiple measures, including a studentdearning 
measure. However, much is left to the district to determine: how decisions will be made, 
what measures will be used, who will conduct the evaluations, and how the evaluation 
process will be managed and supported. The district has determined, in keeping with state 
guidelines, that the new evaluation will assess teachers’ professional practice and their 
impact on student learning. The district leadership would like the system to be supported 
by teachers, to effectively differentiate among teachers, to support teachers’ ongoing pro- 
fessional growth, to lead to improvements in teacher practice, and ultimately to positively 
influence student learning. 

Professional development initiative for science teachers 

A state department of education is implementing a new professional development program 
for science teachers in secondary schools that focuses on promoting inquiry-based 
approaches. The state will partner with informal science partners, such as museums, and 
with industry and will develop a series of content courses for teachers. The professional 
development will use research-based materials and practices so that teachers can build a 
deep foundation of knowledge that will allow them to grow professionally. In addition, 
districts will create professional learning communities and ongoing science instruction 
development opportunities to support inquiry approaches. There will also be profession- 
al development offerings that focus on leadership development to support coaches, lead 
teachers and principals, and curriculum directors in science learning. The goal of this 
program is to enhance teachers’ knowledge in science, increase student participation in 
science courses, and increase student learning and achievement. 

Universal prekindergarten policy 

Massachusetts has implemented a policy to provide prekindergarten education to all chil- 
dren through a blend of private and public providers. According to Early Education for All, 
70 percent of three- and four-year-olds in Massachusetts attend a formal early education 
and care program. The challenge is to ensure that the programs they attend are high- 
quality ones, because as decades of research document, only high-quality prekindergarten 
delivers lasting benefits, particularly for low-income children. 
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Review of logic models 


Supporting materials for this section: slides 49-52. 

Here are a few quick reminders about what a logic model is and what it is not. 

A logic model is: 

• A graphic representation of the theory of change driving a program or policy. 

• A framework for planning, implementation, monitoring, and evaluation. 

A logic model is not: 

• A strategic or fully developed plan for designing or managing a program or policy. 

• An evaluation design or evaluation method. 

While a logic model is not a strategic or fully developed plan or an evaluation design or 
evaluation method, it can be useful in developing any of these more detailed resources. 
The focus of session II of the workshop is on the latter: How does a logic model support the 
development of an evaluation plan for a program or policy? 

If participants have created draft logic models for a program or policy they are engaged in 
or considering in their work, the logic model drafts will serve as the template to guide their 
work throughout this session. If not, they may use the College Ready case logic model on 
slide 51 (or see the two example logic models, one for the College Ready example and one 
for an educator evaluation system, in appendixes C and D). 

Consider the following questions when evaluating a draft logic model: 

• What elements of the logic model were hardest to develop? 

• Is the problem statement the right “grain size”? 

• Within the strategies and activities, did you identify overarching strategies? 

• What assumptions did you uncover? 

• What is the timeframe for your outcomes? 

• What are the impacts? 

• What was your process for developing the model? 

• What requires further explanation or discussion? 
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As an extension of the basic workshop, if the participants come to the 
workshop with draft logic models, the facilitator can ask participants to 
consider the questions provided above as they relate to their own draft 
logic models. Participants can review the questions and brainstorm their 
answers individually. This may be done in a virtual or face-to-face context. 
This will add 15 minutes to the overall time for session II. 


In a face-to-face context, if the group is small, have a participant share one sample 
logic model, and walk the group through the model. If the workshop is a large face-to- 
face gathering, divide participants into smaller groups to discuss the questions above 
as they relate to their own logic models, and assign a facilitator and a note-taker to 
each group. Have the small groups bring back summarizing questions and comments 
to the large group. Sharing logic model samples and the associated discussion will add 
20 minutes to session II. 


Reviewing participants’ logic models will work only if this session is at least a week 
after session I — to allow participants to generate a draft logic model to use during this 
session. 
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Introducing evaluation 


Supporting materials for this section: slides 53-57. 

Program and policy evaluation helps answer important questions that inform this work. At 
a basic level, evaluation answers the questions: Are we successful? Have we had an impact? 
What are the most influential aspects of the program? 

More specifically, evaluations ask questions such as: 

• Is the program or policy effective? 

• Is the program or policy working as intended? 

• What aspects of the program are working? What aspects are not working? 

High-quality evaluation is designed to support your work, inform what you do, and 
enhance your impact. To do so, evaluation should be considered at the onset of program 
and policy design, ideally when the logic model is being developed. In other words, as a 
program or policy is conceived, evaluating the same program or policy should be a part of 
the conversation, by asking questions such as: 

• What do we anticipate to be the impact of this policy? 

• How will we know if we are successful? 

• What do we think will be the most influential aspects of the program? 

All these questions suggest directions for evaluation. Do not wait until the program or 
policy is in the midst of implementation to begin to consider these questions and how to 
answer them. Invest early in considering these questions and designing an evaluation that 
will help answer them. It may also be helpful to involve others, including staff and partici- 
pants, in helping plan the evaluation. 


Activity II. 1: How will you know? 

Consider your own program or policy logic model. How will you know if one or more of your strat- 
egies have been successful? Brainstorm some ways you will know your efforts have yielded the 
results you hope to achieve. 
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For a face-to-face workshop, ask participants to turn to the person next to 
them and answer the question, “How will I know?” Allow each person to 
talk for two minutes. You can then ask for a few volunteers to report out to 
the whole group on their answer. For a virtual workshop, allow people to 
answer the question in the chat pod on the virtual platform. This activity 
will add 10 minutes to the agenda. 


Evaluations generally have one of two purposes: 

• Improve: These are formative, process, or implementation evaluations. 

• Prove: These are summative, results, or outcome evaluations. 

This workbook refers to these evaluation purposes as formative (improve) and summative 
(prove). Most evaluation questions emerge from the strategies and outcomes sections of the 
logic models. You want to know about the strategies that you are trying and how they are 
going, and you want to know about outcomes and impact. 

Generally, evaluations that focus on strategies (and outputs) are formative or process evalu- 
ations, or evaluations that are designed to help guide changes or improvements. Evaluations 
that focus on the outcomes in the logic model are generally summative evaluations — 
designed to prove the value, merit, or impact of the program or policy. 

There are generally four types of evaluations: 

• Needs assessment (formative). This type of evaluation determines what is needed 
(at the onset) and helps set priorities (for example, is more money needed to 
support blended learning?). These types of evaluations are often designed to help 
create or build a program or policy, so a logic model might be developed after the 
needs assessment. In fact, the needs assessment might provide information that 
helps clarify the problem to which the program or policy is designed to respond. 

• Process evaluation (formative). This type of evaluation examines what goes on 
while a program is in progress. The evaluation assesses what the program is, how it 
is working, whom it is reaching, and how (for example, are participants attending 
as anticipated?). 

• Outcome evaluation (summative). This type of evaluation determines the results 
from and consequences of a program, generally for the people most directly affect- 
ed by the program (for example, did participants increase their knowledge or 
change their attitudes or behavior?). 

• Impact evaluation (summative). This type of evaluation determines the net causal 
effects of the program beyond its immediate results. Impact evaluation often 
involves comparing what appeared after the program with what would have 
appeared without the program. These evaluations generally include comparison 
groups, interrupted time series, or other designs that allow evaluators to capture 
what happened to the target compared with what would have happened without 
the program (for example, achievement scores and acceptance rates). 
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Moving from logic model to evaluation questions 


Supporting materials for this section: slides 58-63. 

The purpose of this section is to make the connection between the logic model and devel- 
opment of appropriate evaluation questions, using the logic model as a basis for developing 
the questions. The first step in making the transition from the logic model to a potential 
evaluation is to consider the questions that are derived from the model that you may want 
answered. 

Developing evaluation questions 

As noted in the previous section, some questions ask about improvements to the program 
or policy (formative, process, and implementation questions), while others ask about the 
impacts (summative, results, and outcome questions). Generally: 

• Formative questions (improve) are asked while the program is operating and are 
for the purpose of program improvement or midcourse correction. 

• Summative questions (prove) are asked at completion of or after the program and 
are for the purpose of determining results and assessing effectiveness. 

Regardless of the type of questions, there are some guiding questions to consider for all 
evaluation questions. 

• Can the questions be answered given the program? One of the main reasons for 
building a logic model as part of program evaluation is to determine what ques- 
tions are appropriate based on the program. By describing what the program is, 
the logic model helps determine what is appropriate to evaluate. 

• Are the questions high-priority? Try to distinguish between what you need to 
know and what might merely be nice to know. What are the key, most important 
questions? For whom? Why? 

• Are the questions practical and appropriate to the capacity you have to answer 
them? Consider time, resources, and the availability of assistance needed to answer 
the questions. As appropriate, bring stakeholders together and negotiate a practi- 
cal set of questions. Remember, it is better to answer a few questions thoroughly 
and well. 

• Are the questions clear and jargon-free? Apply the “Great Aunt Lucy test.” Would 
someone like your Great Aunt Lucy or anyone who is not steeped in the language 
of your particular field understand the question? Avoid the use of jargon or vague 
words that can have multiple meanings. Always define key terms so that everyone 
understands the meaning. 
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Activity 11.2: Formative and summative evaluation 

Come up with a formative and summative evaluation question for one of the sample cases, 
such as the blended learning case, or for a program or policy from your own work. 

• Formative evaluation: 

Topic: Blended learning or 

Question: 


• Summative evaluation 
Topic: Blended learning or 
Question: 



For a virtual workshop, if the software functionality allows, have partici- 
pants enter their sample questions in one of two chat pods. Label one chat 
pod “Formative questions” and the other “Summative questions.” 


For a face-to-face workshop, have participants fill in examples in their own work- 
books, and take some examples from the group to discuss. These activities will add 5-7 
minutes to the overall time for session II. 


Considering the audience 

Another key aspect of developing good evaluation questions is considering different 
audiences, or the different stakeholders for a program and policy, the different types of 
questions they might have, and how they would use the answers to these questions (for 
example, what decisions would result from answers). 

Table 1 outlines some traditional audiences, the types of questions they are likely to have, 
and how they might apply answers to these questions to make decisions. 
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Table 1. Traditional audiences, evaluation questions, and uses 

1 Audience 

Typical questions 

Evaluation use I 

Program staff 

Are we reaching our target 
population (for example, high 
school students; low-income 
families with preschool-age 
children)? 

Are participants in the program 
engaged? Satisfied? 

Is the program being run well? 

Flow can we improve the program? 

Day-to-day program operations; 
changes in program design and 
delivery 

Participants 

Is the program helping people like 
me? 

Flow could the program better 
serve my needs? 

Flow could 1 get more out of the 
program? 

Decisions about participation or 
value to them 

Public officials 

Who does the program serve? 

Is it reaching the target 
population? 

What difference does the program 
make? 

Are participants engaged and 
satisfied with the program? 

Is the program cost-effective? 

Decisions about support, 
commitment, funding, scale-up, 
and duplication 

Funders 

Is the program meeting its goals? 
Is the program worth the cost? 

Decisions about ongoing funding; 
accountability 

Source: W.K. Kellogg Foundation, 2006. 
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Activity 11.3: Generating questions for different audiences 

Think about your own context and consider: 

• Audience: Who are the different members of each stakeholder group (staff, participants, 
and the like)? 

• Questions: What questions might different stakeholders have about the program or policy? 

• Evaluation use: How might these different stakeholders use the answers to these 
questions? 


Audience Questions Evaluation use 


Program staff 


Participants 


Public officials 


Funders 



For a face-to-face workshop, have participants work individually on the 
chart for 10 minutes. Then have them gather in groups of three to share 
an example from each audience category. Ask the entire group for several 
examples to talk through how different audiences might inform the types 
of questions developed. 


For a virtual workshop, if the software functionality allows, choose one audience cate- 
gory and have participants enter in the chat pod sample questions that they think this 
particular audience may have. Then in a second chat pod have participants enter their 
ideas about how that particular group of stakeholders may use those questions in an 
evaluation. This activity will add 10 minutes to the overall time for session II. 



It may be wise to pause and take questions at this point in the workshop. 
Slide 64 reflects that pause in the workshop. 
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Generating indicators 


Supporting materials for this section: slides 65-67. 


Activity 11.4: Generating indicators of the flu 

How do we know a child has the flu? 

Take a moment to brainstorm how we know a child has the flu: 


Perhaps we feel her forehead for fever, listen to her sniffles, notice her lethargy, recognize 
a drop in appetite, and eventually take her temperature. All of these are indicators of the 
flu. They do not mean absolutely that the child has the flu, but they do provide specific and 
measurable evidence that suggest the flu. 

In developing evaluation questions, you must consider how you will know that you have 
achieved the program or policy goals and therefore answered the questions of interest. 
This section demonstrates how the logic model can support generation of good indicators 
of program or policy success. 

If asking “Is the program successful?”, some understanding is needed of how “success” is 
measured. In other words, the question, “How will we know the program is successful?” has 
to be answered. The logic model provides some support for this. 

Starting with the logic model’s outputs and outcomes, indicators can be developed that 
answer the question, “How will we know the program is successful?” Indicators are differ- 
ent from the outputs or outcomes included in the logic model: While the outputs or the 
outcomes are more general goals for program implementation or outcomes, indicators are 
specific, measurable targets related to the outcomes of interest. 

In short, indicators are: 

• Specific, measurable targets. 

• Seen, heard, read, and felt. 

• Connected to strategies, activities, outputs, and outcomes. 

• Evidence representing phenomenon of interest (such as the outcome). 

For example, if the outcome is increased parent engagement, the indicator is a specific per- 
centage of parents engaged or a specific increase in number of parents engaged. It is these 
measurable indicators that lead eventually to an answer to the question, “Is the program 
successful?” 
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Indicators do not absolutely mean that a policy or program is responsible for the results we 
measure. To use an example from the College Ready case, a parent engagement program 
(the program) may not be responsible for the rise in college applications among student 
participants (the indicator). There might be other factors, such as a decline in college costs 
or a particularly influential teacher at the school encouraging and supporting applications, 
that lead to an increase in the number of college applications submitted — rather than the 
increase being a direct result of the program. But this increase in college applications (the 
indicator) could reasonably be attributed to a program that works with the students and 
their parents to support college readiness. 

Using the logic model to generate indicators 

Supporting materials for this section: slides 68-76. 

Just as the logic model follows the basic format from inputs (resources) to outputs (strate- 
gies or activities) to outcomes (from short term to long term to impact), this same logic is 
used to generate indicators. 

As stated above, indicators are related to the logic model categories of resources, strategies 
or activities, and outcomes or impact. They go a step further and provide clear numbers 
or percentages, when appropriate, associated with these resources, activities, outputs, or 
outcomes — or measurable evidence of the phenomenon of interest represented by the 
outputs or outcomes. 

• Indicators related to inputs provide information about the resources used, the 
timeliness of the resources, and the relevance of the resources (whether tangible 
or intangible). Indicators related to these inputs may help answer questions about 
impediments or facilitators of implementation. 

• Indicators related to outputs capture the numbers or percentages of workshops 
presented, the numbers of participants, and other data that provide information 
about whether the program was implemented as intended. Did it do what it set out 
to do? Did it reach the right people? 

• Indicators related to outcomes or impacts provide data about the results of partic- 
ipation, such as changes in knowledge, skill, behavior, and attitudes among indi- 
viduals or groups targeted by the program or policy. 

For example, if the strategy or activity was to deliver a parent education class, an indicator 
related to that activity might be the number of classes delivered or the number of parents 
who attended (indicators related to the activities and outputs, or process indicators). If an 
outcome is increased parent understanding of the college application process, an indicator 
would be the number or percentage of parents reporting increased understanding (an indi- 
cator related to the outcomes, or an outcome indicator). 

When generating indicators based on the various elements of the logic model (inputs, 
outputs, strategies and activities, and outcomes), ask these basic questions: 

• What would achieving the goal reflected in the outcome look like? 

• How would we know if we achieved it? 

• If I were visiting the program, what would I see, hear, or read that would tell me 
that the program is doing what it intends? 
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Activity 11.5: Process and outcome indicators 

Using the College Ready program case, this table is designed to help you map a path from an 
activity in the logic model to an output to an outcome to an indicator. 


Activity 

Deliver a set of parent workshops for college readiness. 

Output 

Six workshops developed and delivered; 100 parents recruited to participate. 

Outcome 

Parents increase their understanding of college application process. 

Indicator 

Process: 

Outcome: 



If this is a face-to-face workshop, and participants are prepared to do so, ask 
the participants to start their own table of indicators for their own logic 
model. They will map a path from a single activity in their logic model, to 
an output, to an outcome, to an indicator. Allow an additional 20 minutes 
for this activity. 


Identifying the right indicators 

Supporting materials for this section: slides 77-83. 

Different indicators are related to different types of questions. 

• To know whether the program has sufficient resources or funding to operate, look 
at indicators related to the program inputs. 

• To know whether the program was implemented as intended, look at indicators 
related to the strategies and activities or outputs. 

• To learn the ultimate value and impact of the program, look at outcome-related 
indicators. 
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Some indicators may be more straightforward or easier to measure than others. Sometimes 
one indicator is all that is needed for a clear explanation. For example, school graduation 
rate might be the agreed on indicator for the outcome of decreasing the school dropout 
rate. In other cases, more than one indicator is necessary to capture a more complex 
outcome. 

If the outcome of interest is improved parent involvement in school, several indicators may 
be necessary, such as: 

• Attendance at school meetings. 

• Participation in parent-school organization. 

• Parent calls made to the school. 

• Attendance at school functions. 



For a virtual workshop, if the software functionality allows, ask participants 
to consider what a range of indicators might be to capture the outcome of 
increased parent involvement in school. Flave a chat pod open and have 
participants brainstorm some of the possible indicators. End by going back 
to the presentation and showing slide 87, which contains some of the indi- 
cators listed. 


For a face-to-face workshop, have participants brainstorm with a person next to them 
some of the possible indicators for parent involvement. Then show some examples of 
potential indicators on slide 87. This activity will add 5 minutes to the workshop. 


Quantitative and qualitative indicators 

Supporting materials for this section: slides 84-87. 

Indicators may be quantitative or qualitative. Given the current interest in and demand 
for measurable outcomes, evaluation questions often focus only on the outcomes in a logic 
model and only on the quantitative measures associated with those outcomes. However, 
to attribute quantitative outcomes (such as graduation rates or improvements on standard- 
ized tests) to a program, it is important to ask questions about the process that may have 
contributed to those outcomes. There are some questions that are best answered with a 
mixture of both quantitative and qualitative data. This suggests the need for both qualita- 
tive and quantitative indicators to answer questions of interest. 

• Quantitative data are generally best suited to summative evaluations — such as 
information related to proving the value or impact of the program or policy. 

• Qualitative data are generally better suited to formative evaluations — such as 
those that focus on how to improve the program. 

However, this is not to suggest that all formative evaluations should be qualitative and 
that all summative evaluations should be quantitative. Often, a mix of measures is the best 
approach. Qualitative data, collected through interviews, observations, and other methods, 
often provide the depth of information needed to interpret quantitative data, such as test 
scores or graduation rates. We often want to know both whether a program or policy had 
the desired impact and how. Thus, a mix of measures is advisable. 
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It is a good idea to collect different types of data (for example, quantitative and qualitative) 
from several sources (for example, surveys, interviews, grades) and from different groups 
of stakeholders (for example, students, parents, mentors, staff, partners such as schools). 
While some funders may prefer quantitative data on outcomes, others, such as staff or 
parents, may prefer qualitative data from parents or students. 

For example, in the College Ready case, to learn whether the program increased student 
interest in college, the indicators might include both quantitative (number of applications 
completed) and qualitative data (guidance counselors’ reports on student interest). 


o 


If you have additional time and the participants are prepared, you may 
extend this activity with the task below. If you include the task below, give 
the participants time to generate examples. Then ask for a few participants 
to share their ideas, either in the chat function for a virtual workshop, if 
the software functionality allows, or in a face-to-face workshop. This activ- 
ity will add 10 minutes to the workshop. 


Consider your own programs. Can you think of both quantitative and qualitative indica- 
tors of success? Go back to the evaluation questions you generated for different audiences 
earlier in this workshop, identify two questions, and brainstorm both quantitative and 
qualitative indicators for each question. 

Evaluation question: 


Quantitative indicator 


Qualitative indicator 


Evaluation question: 


Quantitative indicator 


Qualitative indicator 


A few final considerations about indicators. Indicators may: 

• Match the outcomes of interest or questions asked. 

• Be singular for a given outcome or question. 

• Be quantitative or qualitative. 

• Vary based on the audience. 
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Building an evaluation design 


Supporting materials for this section: slides 88-91. 

This section provides tools for building an appropriate evaluation design. 

Once you have generated a good logic model, come up with the best evaluation questions, 
and developed what you believe will be the best indicators of program or policy success 
(either for program improvement or to make the case for program impact), you are ready to 
build the evaluation design. 

Determining what type of evaluation is best for you 

The first question to answer about evaluation design is quite basic: Is the purpose of the 
evaluation to examine process elements (formative) or to examine overall impact (summa- 
tive), or is it a hybrid evaluation, with some of each? Answering this question should help 
clarify what to include in the evaluation. 

You should also return to the question of audience. Who is the audience for the evalua- 
tion, what does the audience want to know, and how will the information be used? 

Consider your capacity: Who will conduct the evaluation, using what resources, and 
within what timeframe? 

Finally, assuming there are some financial, time, or other capacity constraints, what are 
your priorities? What do you need to know? 

Identifying appropriate data sources 

When choosing measures for program evaluation, think about the data collection needs as 
well as data collection capacity. 

• Access pre-existing data. Consider collecting from pre-existing sources of data, such 
as school attendance records or items from surveys that a school district already 
requires students to take, that will meet your evaluation needs. 

• Use existing instruments. When data cannot be collected through existing sources, 
consider existing instruments that measure the same concepts that you are looking 
to measure. These instruments may be effective as written, a small tweak may be 
needed, or you may need to adopt only a few items from the entire instrument. It 
may even be possible to add these instruments or items to existing surveys current- 
ly being administered in your school or district. 

Consider the types of data sources that might serve as indicators of success, both for pro- 
cess-related questions and for outcome-related questions. 

The following data sources may be useful for collecting process and outcome data: 

• Administrative data (program documents, activity logs, registration records, and 
the like). 

• Focus groups. 

• Interviews. 
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• Observations. 

• Surveys. 

• Student test scores and grades. 

• Teacher assessments. 


Activity 11.6: Consider data sources 

Consider the data sources available to you in your program. Brainstorm what relevant data 
sources you already collect, either because they are required or because you choose to collect 
them: 



For a face-to-face workshop, ask for examples from the group. For a virtual 
workshop, ask people to enter examples into the chat pod, if the software 
functionality allows. This will add 5 minutes to the workshop. 
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If there is additional time, you may also include the following activity as 
an extension to the data source brainstorm. Ask participants to take the 
initial brainstorm of data sources and map them onto outcomes of interest 
or to evaluation questions they have already generated or might continue 
to generate through this activity. You may include the following organizing 
chart for their use. This will add 15-20 minutes to the workshop. 


1 Evaluation question 

Outcome of interest 

Data source | 

Is the program reaching the parents who 
are the target of the initiative? 

Parent involvement 

Parent attendance log 


Creating a data collection framework 

Supporting materials for this section: slides 92-94. 

In addition to identifying potential data sources, the details of data collection must be 
ironed out early in planning for the evaluation. It is important to develop a clear sense of 
what types of data are available, who collects and manages the data, and when the data 
are collected. Developing a data collection framework linked to the activities and out- 
comes of interest will help guide the evaluation process. In some cases, the data collection 
framework may look different for process evaluation and outcome evaluation. However, in 
many cases an evaluation will include some process, or formative, and some outcome, or 
summative, components. As such, these may be combined into one framework, as in the 
table below. 
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Case: College Ready program 


Strategy 
or activity 

Output or 
outcome 

Formative 

Summative 

Indicator 

Data source 

Data 

collection 

instrument 

When 

collected 

By whom 

Parent 

education 

strategy 

High rate of 
parent at- 
tendance at 
workshops 

✓ 


70 percent 
of parents 
attend five 
out of six 
workshops 

Administrative 

data 

Attendance 
log at work- 
shops 

At beginning 
of each 
session 

Program 

director 

Parent 

education 

strategy 

Increased 
parent under- 
standing 
of college 
application 
process 


✓ 

85 percent of 
parents who 
attend more 
than four 
workshops re- 
port increased 
understanding 

Parent feed- 
back 

Survey and 
interviews 

Beginning 
of program, 
end of pro- 
gram 

Program 

staff 


It may be useful to distinguish among short-term, long-term, and impact data when creat- 
ing an outcome-specific data collection table (see the table below). The reason for doing 
this may be relevant to only some evaluations, depending on the depth and duration of the 
evaluation plan. However, distinguishing among outcomes in the logic model should help 
guide the types of data that ought to be collected (and when) to reflect those outcomes. 
For example, if changes in student test scores are not anticipated until a program has been 
up and running for three to five years, these data should not be collected (except to serve 
as a baseline, or point from which change will be measured) until the long-term phase of 
data collection. 


Strategy or 
activity 

Output or 
outcome 

Indicator 

Data source 

Data collection 
instrument 

When collected 

By whom 

Short term 

Parent education 
strategy 

Increased parent 
understanding of 
college applica- 
tion process 

85 percent of 
parents who 
attend more than 
four workshops 
report increased 
understanding 

Parent feedback 

Survey and inter- 
views 

Beginning of 
program, end of 
program 

Program staff 

Long term 

Student education 
strategy 

Increased student 
understanding of 
college applica- 
tion process 

80 percent of stu- 
dents who attend 
the workshops ap- 
ply to college and 
get accepted to at 
least one college 
or university 

Student feedback 

Survey and inter- 
views 

End of program 

Program staff 


Impact 
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To extend the discussion of these tables, ask participants to consider what 
long-term and impact rows might look like for the same College Ready 
case. For a face-to-face workshop, take some examples from the group and 
walk through them together, filling in the chart on the board. For a virtual 
workshop, prompt participants to focus first on the long-term row. Just 
focus on the strategy, the output or outcome, the data source, and the data 
collection instrument. This would all be speculative but will help clarify 
the idea of systematically thinking through the data issues to consider. 



It is unlikely that participants will be able to complete the activity below 
during the workshop. Instead, for a face-to-face workshop where partici- 
pants have a common program or policy, suggest that participants work 
in pairs to generate a short-term, a long-term, and an impact example. If 
time is limited, assign pairs to one of the levels (short term, long term, or 
impact). Take a few volunteers to share examples. For a virtual workshop, 
invite participants to select one of the levels (short term, long term, or 
impact) and type an example into the chat pod. It is often helpful to have 
a few examples ready to include in the chat to get people thinking and 
encourage their participation. 


Additional activity: Data collection framework 


Try it yourself. Consider your program or policy and generate a data collection strategy. This activity is best complet- 
ed after having developed a full logic model. 


Strategy or 
activity 

Output or 
outcome 

Indicator 

Data source 

Data collection 

instrument When collected By whom 

Short term 


Long term 


Impact 
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Putting it all together 


Supporting materials for this section: slides 95-97. 

Now that you have a logic model, a set of measurable indicators, some evaluation ques- 
tions, a data collection framework, and at least some idea about evaluation design, you are 
nearly ready to proceed with evaluation. What you still need is the overall evaluation plan 
and timeline. This section provides some tools to help you develop your plan and timeline. 

At this point, you have learned about all the steps necessary to get you to the evaluation. 
In brief, you have learned about how to: 

• Develop a logic model, in collaboration with stakeholders. 

• Clarify who the audience is for the evaluation and how it will be used. 

• Identify and prioritize evaluation questions, based on the logic model. 

• Select appropriate indicators, based on the outcomes of interest. 

• Identify data sources and a data collection plan (including considering whether 
the evaluation is serving a formative/process or a summative/outcome goal). 

• Consider evaluation design, with awareness of resources, capacity, and timeline. 

Once you have completed all these steps, it is still a good idea to develop an evaluation 
prospectus that lays out much of this information in clear, narrative form. 

Evaluation prospectus 

If you are going to look for an external evaluator, or even if you are going to do the eval- 
uation in house, it is wise to have an evaluation prospectus to accompany all the other 
materials you are generating. This prospectus should provide a clear and straightforward 
answer to the following questions: 

• What are you going to evaluate? 

• What is the purpose of the evaluation? 

• How will the results of the evaluation be used? 

• What specific questions will the evaluation answer? 

• What data sources will be necessary to answer these questions? 

• How will the data be analyzed (evaluation design)? 

• What resources are needed to conduct this evaluation? 

• What is the timeline for the evaluation? 

• How will the results be shared or disseminated? 

• Who will manage the evaluation? 
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Evaluation timeline: Example of Gantt chart 


One more tool may be helpful in implementing the evaluation plan. Evaluators often work 
with Gantt charts, which are a kind of timeline, displayed in such a way that readers can 
immediately see a proportionate, chronological account of the time for each evaluation 
task. They can be helpful for clarifying the sequence of activities in an evaluation, moni- 
toring progress toward goals, and communicating with others about the products associat- 
ed with the evaluation. 

• The vertical axis shows the tasks to be completed. 

• The horizontal axis shows the time scale. 

• A shaded area indicates how long each task will take. 

• Milestones are denoted with a ★. 


Task January February March April May June July 


Develop survey 


Select sample 


Administer survey 


Analyze survey data 


Compare survey data 
to indicators 


Write up findings 


☆ 
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Review 


Supporting materials for this section: slides 98-99. 

When introducing logic models, session I noted that they are a useful tool for program 
design, implementation, and evaluation. They should be living documents that are referred 
to throughout the life of the program and the evaluation and amended as needed. They 
are also helpful for guiding a program as it evolves and for ensuring that the work of the 
program remains focused on the key goals and outcomes. 

Logic models are useful for program evaluation, especially when evaluation is designed 
in concert with the logic model. It is much better to consider evaluation at the outset of 
a program’s or policy’s development than as an afterthought or halfway through program 
implementation. 

Finally, engaging key voices — staff, parents, students, funders, and others — in discussions 
about program design and evaluation will promote the buy-in and ongoing support of these 
participants as well as increase the authenticity of the model and the evaluation. 



For a virtual workshop, have participants list one thing they learned or 
will take back to their colleagues in a separate chat pod. For a face-to-face 
workshop, have participants share their ideas and write them on a white- 
board or large notepad paper. 
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Appendix A. Simple logic model 


Problem statement 


Resources 


Strategies 


Outputs 


Short-term 


Long-term 

(inputs) 


and activities 



outcomes 


outcomes 


Impacts 


Assumptions 


A-l 










Appendix B. Template for simple logic model 


Problem statement: 


Resources 

Strategies and activities 

Outputs 

Short-term outcomes 

Long term outcomes 

Impacts 

What resources are or could 
reasonably be available? 

What will the activities, 
events, and such be? 

What are the initial products 
of these activities? 

What changes are expected 
in the short term? 

What changes do you want 
to occur after the initial 
outcomes? 

What are the anticipated 
changes over the long haul? 


CD 

p 


Assumptions: 




Appendix C. Sample logic model for college readiness program 


Problem statement: Low-income high school students in selected communities attend college at a lower rate than their middle-class peers, leading to more limited opportunities, higher rates 
of unemployment, and lower earnings. 


Resources 

Strategies and activities 

Outputs 

Short-term outcomes 

Long term outcomes 

Impacts 

What resources are or could 
reasonably be available? 

What will the activities, 
events, and such be? 

What are the initial products 
of these activities? 

What changes are expected 
in the short term? 

What changes do you want 
to occur after the initial 
outcomes? 

What are the anticipated 
changes over the long haul? 

Partnership with three 

• Establish local college 

• Recruited adequate num- 

Participating students ap- 

■ Participating students are 

■ Low-income students in 

public high schools. 

mentorship program. 

ber of mentors for student 

ply to at least one college 

accepted to and attend 

participating communi- 

■ Community mentors. 

Recruit and prepare peer 

cohort. 

on time. 

college, remaining enrolled 

ties attend college at the 

• Local university space for 

mentors. 

• Developed and delivered 

Parents report increased 

into the third semester of 

same rate as middle-class 

parent meetings. 

■ Develop and deliver 

12 workshops on college 

understanding of the 

college. 

peers. 

■ Volunteer college 

student readiness program 

application process, SAT/ 

college application 

• Participating students’ 

• Low-income students in 

admissions directors for 

(workshops). 

ACT, FAFSA, and college 

process. 

GPAs are above 3.0 at 

participating communities 

application workshop. 

■ Develop and deliver parent 

life. 

• Students report increased 

college, into the third 

graduate from college at 

• Student volunteers 
for childcare at parent 
meetings. 

education (workshops). 

• Developed and delivered 
six workshops for parents. 

• High interest and 
attendance demonstrated 
at all workshops for 
parents and students. 

readiness for college. 
Participating students 
complete FAFSA forms on 
time. 

semester. 

■ Increased parental 
engagement in 
participating high schools’ 
student education. 

the same rate as middle- 
class peers. 

Participating high schools 
see increase in parent and 
student engagement. 
Participating high schools 
state test scores increase 
by x percent. 


Assumptions: College attendance is desired goal for participating communities; high school leaders will remain consistent and support program; parents will show interest and participate in 
program. 




Appendix D. Example of a logic model for an 
educator evaluation system theory of action 


Context: Implement four pillars of effective teaching: teacher evaluation, teacher preparation, teacher induction, and 
professional development. 


Long-term goa/s: To support teachers’ continuous improvement and to improve student learning and other outcomes. 


Evaluation goat: To evaluate the implementation of the state teacher evaluation framework in a subset of schools and districts in 2012/13. 


Strategies 

Participants 

Implementation 

outcomes 

Intermediate 

outcomes 

impact and long-term 
outcomes 

Design and imple- 

Teachers 

District teacher 

Based on infor- 

Alignment of evalua- 

ment new teacher 

Classroom and school 

evaluation plans 

mation gained 

tion and professional 


evaluation systems 
that employ multiple 
measures including: 
Teacher 
observations. 
Evaluator 
and teacher 
conferences. 

• Student-learning 
objectives. 

■ Professional 
growth plans that 
place teachers in 
‘‘tracks” based on 
experience. 

Multiple rating 
categories. 
Teaching standards 
based on the Dan- 
ielson framework. 3 


specialists 

School and district 
administrators 

Department of 
Education 

Students 

Parents 

Other stakeholders 
(such as school 
committees, school 
board) 





correspond with state 
guidelines. 

District teacher eval- 
uation systems have 
been implemented as 
designed with regard 
to multiple measures 
of student learning 
and teacher perfor- 
mance including: 
Teacher 
observations. 
Evaluator 
and teacher 
conferences. 

• Student-learning 
objectives. 

• Professional growth 
plans. 


from evaluation of 
implementation: 
Program changes 
and modifications 
to framework for 
teacher evaluation. 

• Department of 
Education provides 
guidance, training, 
and support materi- 
als for districts. 

• Individual district 
modifications to 
teacher evaluation 
systems. 

• Systems differen- 
tiate among teach- 
ers based on level 
of experience and 
performance. 


support systems 

Improvement in teach- 
er practice. 

Improvement in 
student-learning 
outcomes, including: 
Higher achievement 
on standardized 
assessments. 
Decreased dropout 
rate. 


District teacher 
evaluation systems 
have been implement- 
ed with fidelity with 
regard to use of multi- 
ple rating categories. 


a. Danielson, C. (2007). Enhancing professional practice: A framework for teaching. Alexandria, VA: Association for Supervision and 
Curriculum Development. 


Assumptions: The Department of Education assumes that educator evaluation is one of four essential pillars of effective teaching. To 
maximize long-term outcomes, all pillars need to be in place and integrated. The department also assumes that if districts implement 
reformed teacher evaluation systems that map onto the Blueprint for Effective Teaching in New Hampshire (http://www.education. 
nh.gov/teaching/documents/phaselreport.pdf), then district-level systems will lead to long-term outcomes indicated above. It 
hypothesizes that achievement of outcomes will be influenced by professional climate: Schools with a favorable climate will be more 
likely to have successful implementation. 
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Introduction to workshop 


A logic model is a visual representation of a theory of action or program logic guiding 
the design and implementation of a program or policy and can be used as a tool for build' 
ing a relevant evaluation design. The Regional Educational Laboratory (REL) Northeast 
& Islands administered by Education Development Center created this workshop to help 
groups, such as the research alliances affiliated with the 10 RELs, as well as individual 
alliance members, learn about and build logic models to support program designs and evab 
uations. Based on feedback from alliance members, REL Northeast & Islands learned that 
many of its district- and state-based members would like to build their capacity to develop 
logic models for both evaluating their own programs and working more effectively with 
evaluators whom they engage to conduct evaluations on their behalf. 

This workshop provides a primer on logic modeling and demonstrates how to use logic 
models as a tool for program evaluation. The overarching goals of the workshop are to: 

• Introduce logic models as an effective tool for program or policy design, implemen- 
tation, and evaluation. 

• Practice the elements of a logic model. 

• Provide guidance in appropriate steps for building a logic model for a program or 
initiative. 

• Practice using logic models to develop evaluation questions and indicators of 
success. 

• Provide guidance in how to determine the appropriate evaluation for a specific 
program or policy. 

Versions of this workshop were presented to three REL Northeast & Islands research alli- 
ances in 2013 in two different formats. The Puerto Rico Research Alliance for Dropout 
Prevention participated in a three-hour face-to-face workshop focused on supporting the 
alliance’s effort to generate a common vision for dropout prevention work. The Urban 
School Improvement Alliance and the Northeast Educator Effectiveness Research Alli- 
ance both participated in virtual webinars for a broad audience of practitioners interested 
in developing skills and capacity to develop and use logic models to increase individual 
and group capacity to design and evaluate programs. 


The authors thank the following people from Education Development Center, Inc., for their expertise and 
resources in developing these materials: Katrina Bledsoe, Leslie Goodyear, Brian Lord, and Anne Wang. 
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Session I. Learning about logic models (2 hours) 


Agenda 


5 minutes 

Introduction and goals 

Facilitator offers introductions followed by an overview of the agenda and purposes of 
the workshop. 

10 minutes 

Introducing the cases 

Facilitator reviews cases that will be used as examples throughout the workshop. 

Activity 1.1 (7 minutes): Participants brainstorm the goals of the programs in the cases 
and the types of questions that might be appropriate to ask about the cases. 

20 minutes 

What is a logic model? 

Facilitator introduces logic models as a useful tool for program design, implementation, 
and evaluation. Facilitator then introduces three types of logic models with three 
different purposes. Facilitator discusses overall idea of inputs-outputs-outcomes that 
drives logic model development and the logic of logic models. 

Activity 1.2 (10 minutes): Participants complete an inputs-outputs-outcomes table for 
the College Ready case. 

65 minutes 

Elements of a logic model 

Facilitator walks through all elements of a logic model. Several individual elements have 
an associated activity. 

Problem statement: Includes discussion of what a problem statement is and 
the questions that need to be considered in developing an appropriate problem 
statement. 

Activity 1.3 (7 minutes): Participants brainstorm their own problem statements. 

■ Outcomes: Includes discussion of short- to long-term outcomes and impacts, as 
well as review of what outputs are and how they differ from outcomes. Guidance for 
generating outcomes is provided. 

Activity 1.4 (15 minutes): Participants fill in a table that helps generate outcomes for their 
own examples or for one of the case examples. 

■ Strategies and activities: Focuses on program components and how sequencing and 
clustering of activities should be considered. There is no workshop activity associated 
with this element, but there is an activity in the workbook provided as a suggestion for 
participants to do on their own following the workshop. There is a brief pause here for 
questions. 

■ Resources: Discusses material and intangible resources. 

Activity 1.5 (7 minutes): Participants brainstorm at least five nonmonetary resources 
available to them in a program. 

• Assumptions: Discusses examples of assumptions and the importance of being 
explicit about assumptions. 

Activity 1.6 (10 minutes): Participants brainstorm internal and external assumptions for 
their program or initiative. 

10 minutes 

The logic in a logic model 

Facilitator discusses the if-then statements that are embedded within logic models. 
Facilitator then walks participants through the logic of the if-then statements in the 
blended learning case. 

Activity 1.7 (7 minutes): Participants practice with the College Ready case and order a 
series of if-then statements. 

10 minutes 

Next steps 

Facilitator closes with discussion of some next steps for developing a logic model and a 
reminder of how logic models can support program design and evaluation. 

Activity 1.8 (5 minutes): Participants indicate a next step for their work. 
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Pre-assignment 


Consider a program or policy that you lead or are actively involved in designing, imple- 
menting, or evaluating. Come to the session with the following materials or information: 

• For whom is the program or policy designed? 

• What are the main goals or objectives of the program or policy? 

• What is the timeline and duration of the program or policy? 

• What are the major activities or strategies associated with the program or policy? 

Program or policy: 



Main goals 

Timeline and 

Major activities 

For whom 

or objectives 

duration 

or strategies 
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Goals 


Supporting materials for this section: slide 4- 
Session I of the workshop will: 

• Introduce logic models as an effective tool for program and policy design, imple- 
mentation, and evaluation. 

• Invite participants to practice the elements of a logic model. 

• Provide guidance in appropriate steps for building a logic model for a program or 
initiative. 
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Introducing the cases 


Supporting materials for this section: slide 5. 

This section describes the two cases that are used as the examples throughout the work- 
shop (and provides some additional case examples that may be used if more relevant to the 
particular audience). These examples are based on real programs or policies. These cases 
provide a common language for discussing all aspects of the logic model. Participants may 
also want to draw on their pre-assignment, in which they generated their own example, to 
use throughout the workshop. 

College readiness high school program 

College Ready is a schoohbased college access program for students in grades 9-12. Stu- 
dents are identified for the program based on eligibility for free or reduced-price lunch, 
recommendations from school guidance counselors, and recommendations from grade 8 
English and math teachers. Students participate in monthly meetings as a group with the 
College Ready staff, are provided with one-on-one counseling with College Ready staff, 
are assigned an adult mentor and a peer mentor, and participate in a series of evening and 
summer workshops. In addition, families make a commitment to the program and attend a 
series of workshops specifically designed to prepare the whole family for the college appli- 
cation process. The goal of the program is to significantly increase college attendance 
among low-income students. 

Blended learning schools 

An urban district is going to convert a few of its schools into blended learning schools 
(in which students learn both online and face-to-face with a teacher to personalize their 
instruction). The schools will be using the individual rotation model, which allows students 
within a given course or subject to rotate on an individually customized, fixed schedule 
between online courses and a variety of classroom environments with face-to-face teach- 
ers. Individual students have their own netbook and a unique schedule. The model also 
includes site coordinators and a principal who is involved in daily classroom observation. 
The goal of the model is to improve student achievement and individualize instruction. 
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Activity 1.1: Discussion of cases 

• Individually. 

Working on your own for about two minutes and using the worksheet below, consider the 
College Ready or blended learning case above. What are the goals of the program? What might 
we want to know about it? Consider questions of implementation, effectiveness, and impact. 

• In large-group discussion. 

What are your ideas about the goals of the program and what do you want to know about it? 


What are the goals of What do we want to know about 

the program or policy? the program or policy? 
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What is a logic model? 


Supporting materials for this section: slides 6-9. 

This section provides a primer on logic models, including different types of logic models 
and their potential purposes. 

In the most basic terms, logic models provide a kind of map for a program or initiative, 
helping clarify a program or policy’s destination, the pathways toward the destination, and 
markers along the way. 

Consider: 

• Where are you going? 

• How will you get there? 

• What will tell you that you have arrived? 

Logic models provide a simplified picture of the relationships between the program or 
policy inputs (resources, strategies, activities) and the desired outcomes of the program. 

Logic models present a theory of action or change that drives the program or policy and 
makes explicit any assumptions about both the resources at the disposal of the program 
and the rationale behind the effort. 

A logic model is valuable in supporting: 

• Program planning. 

• Program implementation. 

• Program monitoring. 

• Program evaluation. 

Why use a logic model? A logic model: 

• Brings detail to broad goals. 

• Helps identify gaps in program logic and clarify assumptions. 

• Builds understanding and promotes consensus. 

• Makes explicit underlying beliefs. 

• Helps clarify what is appropriate to evaluate and when. 

• Summarizes complex programs for effective communication. 

A logic model is useful in designing program and policy evaluation, because a logic model 
helps clarify both what the program, initiative, or policy is and what it is not. This kind of 
clarification is helpful in building an evaluation design that can capture the program’s or 
policy’s influence and impact. 

What are the limitations of a logic model? A logic model is not: 

• A strategic or fully developed plan for designing or managing a program or policy. 

• An evaluation design or evaluation method. 

While logic models are useful tools for building program plans or evaluation designs, addi- 
tional work is necessary to create both programmatic and evaluation plans. 
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Types of logic models 


Not all logic models are the same, nor are they designed for the same purpose. Just as logic 
models may aid in program design, implementation, and evaluation, the type of model 
developed varies somewhat based on its purpose. There are three main types of logic 
models: 

• Theory approach. 

• Activities approach. 

• Outcomes approach. 

Theory approach models 

Logic models that describe the overall theory of change provide a “big picture” of the 
program and may be useful for program design and overall communication of the program 
theory. These models provide a clear description of why the developers believe the program 
or policy will be effective in achieving the goals. In the blended learning case, a theory 
approach logic model might help clarify the assumptions implicit in the push for alter- 
natives to traditional brick and mortar classrooms and describe the relationship between 
initiating blended learning and the expected outcomes for students who participate. 

Consider: 

• What might be the logic in a theory approach model for your own program or 
policy? (The “big picture” theory of change about your initiative?) 

Activities approach models 

Activities approach models focus on laying out the specific strategies and activities associ- 
ated with a program. These models closely examine the relationship among the activities, 
considering sequence and timing of implementation, as well as how the activities link to 
outcomes. This type of logic model is most useful in program implementation, monitor- 
ing, and management. In the College Ready case, this type of logic model would consid- 
er the elements of the program and how they would be optimally ordered and managed. 
For example, what role would the different mentoring components have? How would they 
relate to one another? In this type of model, relationships among variables are made explic- 
it with arrows, concentric circles, and other graphic representations of relationships. 

Consider: 

• Why consider the sequence and relationship among activities in a logic model? 
How might that help you? 

Outcomes approach models 

Outcomes approach models are most useful for program evaluation. They consider the 
strategies and activities as they relate to the desired results of a program or policy. In these 
models, the focus is on outcomes, and they often divide the outcomes into short-term out- 
comes, long-term outcomes, and impacts. A theory of change drives these models just as it 
does the others. But in an outcomes approach logic model, the emphasis is on examining 
the outcomes and making the case that the program or policy is responsible for the desired 
outcomes. 
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Consider: 

• Why divide outcomes into short term and long term? What is the difference 
between outcomes and impacts? 

Inputs-outputs-outcomes 

Supporting materials for this section: slides 10—14- 

In its simplest form, a logic model is a graphic representation of the relationship among a 
program’s or policy’s inputs (what is invested in the program), the outputs 1 (what is done 
with these investments), and what the outcomes are (what are the results). 

Take a simple example: You have a headache and you want it to go away. 

• What is the input? 

° Quiet time. 

° Water. 

0 A hot compress. 

° Two aspirin. 

• What is the output (that is, what do you do with the inputs)? 

0 Sit quietly for five minutes. 

° Drink a full glass of water. 

° Put hot compress on. 

0 Take aspirin. 

• What is the outcome? 

° You are more relaxed, 
o You are hydrated. 

° Your headache goes away. 

° You are able to return to your work. 

Below is an example from the blended learning case. Some base inputs, outputs, and out' 
comes have been laid out, reflecting the overall logic of the program. 


Inputs Outputs Outcomes 


Existing technology 
infrastructure 

Technology integration staff 
person for three schools 
Teachers' enthusiasm in three 
schools 

Technology integration grant 


Infrastructure audit completed 
Completion of six days of 
summer teacher professional 
development 
Six blended learning 
classrooms established 


Teachers' reported use of 
diverse instruction strategies 
increases 

Student engagement increases 
Student achievement on 
district-wide assessments 
improves 


1. The use of the term “output” here differs from its use when outputs are included as a specific element in 
a complete logic model. The use here is in terms of the general inputs-outputs-outcomes logic of a logic 
model. In this case, outputs do not refer to the measurable, targeted activities that are actually completed; 
rather, they refer more generally to what is done with the inputs to achieve the outcomes — that is, as one 
part of the overall logic embedded in a logic model. While possibly confusing, the main idea here is that a 
logic model sets up a logical progression from resources to activities to outcomes. The section that discuss- 
es the specific elements of a full logic model separates strategies and activities from outputs. 
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Activity 1.2: Inputs-outputs-outcomes 

Using one of the example cases (College Ready), consider the inputs, outputs, and outcomes. 
This activity helps illustrate the basic purpose and premise of logic models, but what is gen- 
erated is not an actual logic model. Rather, this activity is designed to help you understand 
the overall logic undergirding logic models. For more information on the elements of the logic 
model, see the next section. 


Inputs 


Outputs 


Outcomes 
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Elements of a logic model 


Supporting materials for this section: slides 15-18. 

This section provides an overview of — and opportunities to practice — all elements of a 
logic model, which are listed below. The pages that follow delineate these elements and 
use examples from the two cases. The elements are discussed in a different order from their 
presentation in the simple logic model on slide 15 (and in appendix A) because the order 
below makes the most logical sense when working through the different elements of a logic 
model. For example, once a problem statement is determined, thinking about the overall 
goals would be the next logical step, so discussing outcomes should be considered next. 

• Problem statement. 

• Short- and long-term outcomes. 

• Impacts. 

• Outputs. 

• Strategies and activities. 

• Resources (inputs). 

• Assumptions. 

Problem statement 

The problem statement is the problem or challenge that the program or policy is designed 
to address. 

Consider: 

• What is the problem or issue? 

• Why is this a problem? (What causes the problem?) 

• For whom (individual, household, group, community, society in general) does this 
problem exist? 

• Who has a stake in the problem? (Who cares whether it is resolved?) 

• What is known about the problem, issue, or people who are involved? What 
research or experience is available? What do existing research and experience say? 

Finally, ask whether the problem statement is too big or too small. The final problem state- 
ment should be targeted and specific, but it should not be a simple restatement of the 
program as a need. For example, in the blended learning case, “Students lack access to 
their own netbook” is really a statement about the lack of the program. The problem state- 
ment should address the real issues underlying the need for the program, such as “Students 
have limited one-on-one attention from teachers.” 
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Activity 1.3: Problem statement 


Consider the problem statement most appropriate to the challenge you face in your work, 
related to a program you have in place or one you would like to initiate. In other words, consider 
the problem for which your program or policy is the “answer.” Brainstorm key ideas associated 
with the relevant problem. Model your brainstorm after the example below, in terms of brevity. 
There are several different ideas reflected below; in your brainstorm, consider all potential 
ideas or elements of the problem. Remember that a problem statement should be targeted 
and specific. 

Case: Blended learning 

• Students are not actively engaged in their learning. 

• Courses are sometimes monotonous. 

• Students have limited one-on-one attention from adults. 

• Students’ courses are not personalized. 

• Students are all expected to work at the same pace. 

Your brainstorm: 


Outcomes 

Supporting materials for this section: slides 20-27. 

While outcomes are not the next item one sees when one looks from left to right across 
a traditional logic model, they are a logical next step to discuss when examining the ele- 
ments of a logic model. Outcomes should be thought of next because knowing the goal in 
relation to solving the problem defined in the problem statement will help in developing a 
plan for achieving the outcomes. After identifying the problem, it makes sense to identify 
the overall goal, or outcomes, next. 

Outcomes ask, “What difference does it make?” In other words, what is the difference that 
the resources, and strategies and activities, taken together, should have on the various 
participants in these efforts? For example, in the College Ready case, “What difference 
does the mentoring program have on students’ daily school attendance?” or “What dif- 
ference does the mentoring program have on students’ grade point average or college 
attendance?” 
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Outcomes usually come in stages and fall along a continuum from short- to long-term out- 
comes. The language to describe these outcomes varies; this workbook refers to short-term 
outcomes, long-term outcomes, and impact. Other terms include: 

• Short-term: initial, immediate, proximal. 

• Long-term: medium-term, intermediate, midpoint. 

• Impact: long-term, final, ultimate, distal outcome. 

Short-term outcomes 

Short-term outcomes are the most immediate and measurable results for participants 
that can be attributed to strategies and activities. For example, a program that promotes 
increased parent engagement in students’ college planning might have a short-term goal of 
increased parent participation in the provided parent sessions. 

Long-term outcomes 

Long-term outcomes are the more distant, though anticipated, results of participation in 
strategies and activities. When it comes to short- and long-term outcomes, it is good to 
think about the overall timeframe for the program. Sometimes, short term is considered to 
be as short as six months or as long as three years. Long term might be two years or as long 
as six years. The important point is to consider the program and identify the timeframe, 
specific to the initiative, for short- and long-term outcomes. For example, a long-term 
outcome for a program focused on increasing college readiness may be improved academic 
performance of participants in the program. 

Impacts 

Impacts are the desired outcomes of long-term implementation of strategies and activities 
that depend on conditions beyond the program’s scope of strategies. These may be called 
the “blue skies” or the big picture types of objectives that are more distant from the actual 
strategies and activities and less within the control of the program or policy to realize. 
Often these are considered to be 7-10 years after initial implementation. For example, an 
impact of a college readiness program might be an increased percentage of students grad- 
uating from post-secondary institutions after participating in their high school’s college 
readiness program. 

Outputs versus outcomes 

Some logic models include both outputs and outcomes. Outputs differ from outcomes in 
that they capture data about what is done rather than what is expected to be achieved 
as a result of what is done. Outputs can best be described as activity data and are useful 
for tracking program implementation. They often provide detail about the breadth and 
reach of strategies and activities. Outputs capture size and scope; they describe or count 
strategies and activities, such as the number of parent sessions delivered, the program par- 
ticipation rates, the number of materials developed or distributed, and so forth. Using the 
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College Ready program as an example, another way to think about the difference between 
outputs and outcomes is to consider the questions: 

“Is the parent education program being delivered as intended?” (output question) 
versus 

“Are parents who participate in the parent education program becoming more actively 
involved in their children’s education?” Or, a long-term outcome might be, “Is the college 
acceptance rate for participating students increasing?” (outcome question) 

It is important not to confuse outputs for outcomes. A program that is good at delivering 
activities and services may achieve its outputs without achieving its outcomes. Yet, it is the 
outcomes that make the difference in response to the problem identified. 


Activity 1.4: Focus on outcomes 

Being clear about program outcomes is essential for both focused program implementation 
and effective evaluation. The table below is designed to promote a step-by-step approach to 
outcome development. The columns are: 

• Who is the target? Who is the group targeted with the strategy? Is it students? Parents? A 
school? In this example, based on the College Ready case, the target is participating high 
school seniors in three high schools that participate in the program. 

• What is the desired change? Use an action verb to demonstrate a kind of change or an 
impact. For example: increase, improve, engage. 

• In what? What is the activity, strategy, or program in which the target population is going to 
enact this desired change? What is the resulting action in which the target population will 
engage to achieve the goal? 

• By when? Here is where the timeline for outcomes is clarified. Is a particular outcome a 
short- or long-term outcome? 

Enter an example from your own context, related to a program or policy initiative you have 
in place or would like to develop. If you prefer, consider the College Ready or blended learning 
case and fill in the table with a relevant example from one of those cases. 


What is 
the target? 

What is the desired 
change? (action verb) 

In what? 
(results) 

By when? 

Participating high school 

Increase 

Completed and 

By June 2014 

seniors in three urban 


submitted applications 


high schools 


to post-secondary 




institutions 
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Outcomes checklist 


Consider the following criteria when examining outcomes: 

• Are the outcomes important? Do they represent significant changes or improve' 
ments that are valued by participants and key stakeholders? Outcomes may be 
achievable but not really worth the effort. If the outcomes were achieved, would 
anyone care? 

• Are the outcomes reasonable? Are the outcomes connected to one another and 
linked in a reasonable order (from short term to long term to impact)? Is it likely 
that one will lead to the next? Another way to think about this is to consider 
the if-then statements (or logic statements) embedded in a chain of outcomes. 
For example, using the College Ready case, will increased parent participation in 
workshops on college readiness lead to students’ completing more college appli- 
cations? Will access to online courses lead to increased student engagement 
and achievement? Sequence and timing of activities and intended outcomes are 
important to consider. 

• Are the outcomes realistic given the nature of the problem and available resourc- 
es and abilities? Will the program lead to or help contribute to these outcomes? 
(Be careful to ensure that the outcomes are realistic given the level of effort.) In 
other words, is it realistic to expect an increase in student achievement from one 
parent education class? Ask hard questions about the outcomes as they relate to 
the actual program or policy. 

• Are unintentional or possibly negative outcomes being attended to? It is important 
to anticipate and consider the unintended or potentially negative outcomes that 
might result from the set of strategies and activities. What are potential negative 
effects of the program or policy? What else might happen that is different from 
what is intended? How else might the sequence of events unfold? For example, 
could access to online courses lead to lower student attendance? Considering the 
unintended consequences allows program and policy designers to consider how to 
prepare for these possible outcomes and also helps evaluators be attuned to these 
possible consequences in the evaluation design. 

Another common set of criteria for outcomes is the S.M.A.R.T. goals. These are: 

• Specific. 

• Measurable. 

• Action oriented. 

• Realistic. 

• Timed. 

Strategies and activities 

Supporting materials for this section: slides 28-29. 

Strategies and activities are the program components, or the game plan for the program 
or policy. This is an inventory of all the strategies and activities designed to achieve the 
outcomes. However, it is more than a simple listing of activities. There are two questions 
to ask when inventorying the activities, services, products, and events that make up the 
program or policy: 
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• What is the appropriate sequence or order of these activities? 

Consider the College Ready case: It may be important that the mentoring element 
of the program come prior to the delivery of the parent workshop series. Or 
perhaps these activities should be concurrent. Consider the appropriate order of 
activities and how they relate to one another. 

• Are there certain activities that, taken together, add up to a kind of overall strate- 
gy? Do certain activities bundle or cluster together? 

Consider the blended learning case: Perhaps there is a series of training needs 
related to instituting the blended learning model, such as new professional deveh 
opment offerings for teachers, new demands on the technical support staff at the 
schools, and new requirements for paraprofessional support to the classrooms, 
that bundle together as an overarching strategy. Perhaps this is the professional 
training strategy. This may be different from other strategies associated with the 
initiative, such as infrastructure or family engagement. Creating these clusters of 
activities helps streamline the logic model and supports evaluation; the evaluation 
will then assess a set of strategies, rather than individual activities. 

Here is a brief example from the blended learning case, illustrating both the sequence in 
which activities might unfold and how specific activities relate to a set of core strategies. 


1 Activities 

Sequence 

Strategy | 

Develop teacher training materials 

1st 

Professional training 

Deliver summer institute for teachers 

2nd 

Professional training 

Conduct technology audit 

lst a 

Infrastructure 


a. Occurs concurrently with the develop teacher training materials activity and prior to the deliver summer 
institute for teachers activity. 
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Additional activity: Strategies and activities in sequence 

Consider a series of activities that are a part of your own work. List some relevant activities, 
give the sequence or order in which they are supposed to occur, and consider the overarching 
strategy within which these activities fall. In other words, does your chosen program or ini- 
tiative have a core set of strategies that guide the activities, events, programs, and the like 
that you provide? This activity is suggested as something to work on independently after the 
workshop. 


Activities Sequence Strategy 


Resources (inputs) 

Supporting materials for this section: slides 31-33. 

Resources include both the material and the intangible contributions that are or could 
reasonably be expected to be available to address the problem. 

• Material resources include: 
o Money. 

° Materials and equipment. 

• Intangible resources include: 

° People. 

° Time. 

° Partnerships. 
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Activity 1.5: Intangible resources 

Brainstorm intangible resources (example from College Ready case). 

• Community mentors. 

• Local university space for parent meetings. 

• Volunteer college admissions directors for application workshop. 

• Student volunteers for childcare at parent meetings. 

What intangible resources are at your disposal? Brainstorm at least five nonmonetary 
resources that are available to you in a program you operate or manage: 


Assumptions 

Supporting materials for this section: slides 34-36. 

Assumptions are beliefs about participants, staff, and the program, as well as about how 
change or improvement may be realized. Being explicit about assumptions is one of the first 
and most important considerations during program design, implementation, and evaluation. 

Consider the College Ready case: The program assumes that students who participate want 
to go to college and that college enrollment will lead to a better life for participants. Often 
the assumptions embedded in a program or policy are critical to the success or failure of the 
overall initiative. Assumptions may be internal (assumptions about participants, resources, 
and how the program will function) or external (beliefs about how change occurs, values 
embedded in the program, or findings from prior research). 

Recall the headache example from earlier in the workshop. You had a headache, you tried 
a few things to get rid of it (water, aspirin), and you felt better. The outcome was that the 
headache went away. However, between the problem (the headache) and the outcome (no 
headache) were several assumptions. For example, you assumed that you did not have an 
allergy to aspirin, that there was no loud noise persisting in the background, and so forth. 
Clarifying and making explicit the assumptions behind the program, in terms of both the 
specific elements related to implementation and the assumptions embedded in the theory 
driving the initiative, are critical to developing a thoughtful logic model. 

In the blended learning case, internal assumptions might include a belief that the school 
leadership will support the blended learning classrooms going forward and that the staffing 
available will be adequate to support implementation. External assumptions might include 
a belief that access to varied learning modalities will increase student engagement and 
that increased student engagement will yield increased student achievement. These exten 
nal assumptions are both related to the theory of action or change driving the initiative. 
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Activity 1.6: Uncovering internal and external assumptions 

Consider your program or initiative. Brainstorm the range of assumptions embedded in the 
program design and in the overall theory of action driving the initiative. 


Internal assumptions External assumptions 
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The logic in a logic model 


Supporting materials for this section: slides 37-39. 

The purpose of this section is to understand the logic embedded in a logic model and 
recognize the need to identify the possible relationships and connections among various 
elements of a logic model. 

Understanding these if-then relationships is essential to uncovering the theory of action 
or theory of change driving a program or policy. Access to and application of resourc' 
es will lead to programs that reach the target participants, and when these populations 
are reached by such programs, unmet needs will be met and circumstances will change, 
solving the problem that initiated this work. 

Consider the blended learning case: 

• If the district applies funds to support blended learning in three schools, then the 
schools will provide teachers with professional learning opportunities and estab- 
lish the infrastructure to support blended learning. 

• If the schools provide teachers with professional learning opportunities and estab- 
lish infrastructure to support blended learning, then students’ access to online 
courses and varied learning environments will increase. 

• If students have increased access to online courses and varied learning environ- 
ments, then teachers will be able to personalize instruction and the students will 
be more engaged in their learning. 

• If teachers personalize instruction and students are more engaged in their learn- 
ing, then students will be able to master content and develop their skills at a pace 
appropriate to the individual student. 

• If students master content and develop their skills at a pace appropriate to the 
individual student, then they will perform better on standardized assessments of 
their learning. 

The then clause in one statement becomes the if clause in the next statement. This is 
important; when the language changes from then to if, the intention of the statement may 
change. In some logic models, if-then statements are written right into the model to make 
the theory of change explicit. 
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Activity 1.7: If-then statements 


Consider the College Ready case. Move the statements around to make a series of logical 
if-then statements below. Consider the sequencing of events. The statements below include 
strategies and activities, traditional outputs, and outcomes. 

IF THEN/IF 

THEN/IF THEN/IF 

THEN/IF THEN 

1. We develop a series of college readiness workshops for parents. 

2. Parents help their students with the application process. 

3. We recruit parents to participate in the workshops. 

4. Parents better understand the timelines and demands of the college application process. 

5. Students meet financial aid and college application deadlines. 

6. Parents attend the workshops. 
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Next steps 


Supporting materials for this section: slides 40-42. 

In building a logic model, it is important to consider the following questions: 

• Do I understand the different elements of the logic model and how they differ? 

• Who should I consult in developing the logic model? What colleagues and stake- 
holders should be participants in developing the logic model? 

• Who will be responsible for seeing this through? 

• How do I know I have captured the theory of action guiding the program? 

• How will we use the logic model once it is developed? 


Activity 1.8: Next steps 

Consider what your next step might be with regard to logic models. Consider where you are in 
the development of a new program or in an evaluation of a program already in place. How can 
logic models support this work? 
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Final thoughts 


Supporting materials for this section: slides 43-44. 

Here are a few quick reminders about what a logic model is and what it is not. 

A logic model is: 

• A graphic representation of the theory of change driving a program or policy. 

• A framework for planning, implementation, monitoring, and evaluation. 

A logic model is not: 

• A strategic or fully developed plan for designing or managing a program or policy. 

• An evaluation design or evaluation method. 

While a logic model is not a strategic or fully developed plan or an evaluation design or 
evaluation method, it can be useful in developing any of these more detailed resources. 

A logic model is likely to be much more effective, useful, and honest if the process of 
generating the logic model has engaged a broad range of stakeholders during the design 
process. Including key voices such as staff, parents, students, funders, and others in discus- 
sions about program design and evaluation will promote the buy-in and ongoing support of 
these participants as well as increase the authenticity of the model. 

Logic models should be living documents that are referred to throughout the life of the 
program and the evaluation and should be amended as needed. They are also helpful to 
guide a program as it evolves and to ensure that the work of the program remains focused 
on the key goals and outcomes. 

Logic models are useful for program evaluation, especially when evaluation is considered 
in concert with creating the logic model at the early stages of program development. It is 
much better to consider evaluation at the outset of development of a program or policy 
rather than after or halfway through program implementation. 

Good luck with this work, and please contact us with questions! 
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Session II. From logic models to program and policy evaluation (1.5 hours) 



Agenda 

5 minutes 

introduction and goals 

Facilitator reviews the goals of the session and the agenda. 

8 minutes 

Review of logic models 

Facilitator reviews what was learned about logic models, what they are useful for, and 
what limitations they have. A graphic of a simple logic model and a logic model from one 
of the cases will be reviewed. If participants come with their own draft logic models, two 
will be selected for presentation and discussion. 

15 minutes 

Introducing evaluation 

Facilitator introduces the types of questions that evaluation is designed to answer, the 
value of implementing evaluation at the onset of program development, and the role that 
logic models play in supporting evaluation. Two purposes of evaluation are presented: 
evaluation that focuses on improvements and evaluation that focuses on proving or 
demonstrating the impact or outcomes of an initiative. 

Activity II. 1 (7 minutes): Participants brainstorm ways they know their program or policy 
is achieving its goal. 

13 minutes 

Moving from logic model to 
evaluation questions 

Facilitator begins with more about types of evaluation questions, followed by guidelines 
for good questions. Facilitator then introduces the idea of different audiences desiring 
different information about a program or policy and therefore asking different questions. 
Participants will be introduced to a table that delineates different types of audiences, 
questions, and uses of evaluation. 

Activity 11.2 (4 minutes): Participants brainstorm formative or summative evaluation 
questions about their own program or policy or for one of the sample cases. 

Activity 11.3 (4 minutes): Participants practice generating questions for different audiences. 

25 minutes 

Generating indicators 

Facilitator introduces the concept of indicators and provides an overview of how 
indicators are generated from the logic model, specifically from the strategies and 
activities and outcomes sections of the model. Facilitator provides an example of this for 
the College Ready case. Section closes with a discussion of qualitative and quantitative 
indicators and the use and value of both types of measures in an evaluation. 

Activity 11.4 (3 minutes): Participants brainstorm indicators of the flu. 

Activity 11.5 (3 minutes): Participants brainstorm outcome and process indicators based 
on one of the sample cases. 

15 minutes 

Building an evaluation 
design 

Facilitator begins this section with the question, “What type of evaluation is right for 
you?" and suggests that determining the purpose — formative, summative, or hybrid — is 
critical to building an evaluation. Facilitator transitions to more discussion about data 
collection, specifically considering the types of data available to participants. Types of 
data, both quantitative and qualitative, are reviewed. Then facilitator introduces the 
data collection framework tool, which outlines the outcomes of interest, data sources, 
responsible parties, and timeline. This is followed by discussion of evaluation design, as 
distinct from the data collection framework. 

Activity 11.6 (5 minutes): Participants brainstorm their own data sources. 

6 minutes 

Putting it all together 

Facilitator discusses an evaluation prospectus or overview and the key questions to 
consider when generating this short document, which can serve as the “calling card” 
for an evaluation, either for potential funders or for potential evaluators. The facilitator 
closes with a presentation of a Gantt chart as a useful tool for managing an evaluation 
and considering realistic timelines and deliverables. 

3 minutes 

Review 

Facilitator closes workshop with thank you and invitation to be in touch with further 
questions. 
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Pre-assignment 


Based on the work in session I, participants may come to the workshop with a draft logic 
model for a particular program or policy. If participants do not have their own draft logic 
model, they should familiarize themselves with the sample logic models in appendixes C 
and D, as these will be drawn on for examples throughout the workshop. 

A sample logic model template is provided in appendix B and may be used to generate a 
simple logic model. Participants will use this logic model to guide their work in the session. 
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Goals 


Supporting materials for this section: slide 48. 

The purpose of session II is to demonstrate how logic models may be used as a tool specific 
to developing a program or policy evaluation. The session will: 

• Reintroduce logic models as an effective tool, specifically for evaluation. 

• Invite participants to practice using logic models to develop evaluation questions 
and indicators of success. 

• Provide guidance in how to determine the appropriate evaluation for a specific 
program or policy. 
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Example cases revisited 


College readiness high school program 

College Ready is a school-based college access program for students in grades 9-12. Stu- 
dents are identified for the program based on eligibility for free or reduced-price lunch, 
recommendations from school guidance counselors, and recommendations from grade 8 
English and math teachers. Students participate in monthly meetings as a group with the 
College Ready staff, are provided with one-on-one counseling with College Ready staff, 
are assigned an adult mentor and a peer mentor, and participate in a series of evening and 
summer workshops. In addition, families make a commitment to the program and attend a 
series of workshops specifically designed to prepare the whole family for the college appli- 
cation process. The goal of the program is to significantly increase college attendance 
among low-income students. 

Blended learning schools 

An urban district is going to convert a few of its schools into blended learning schools 
(in which students learn both online and face-to-face with a teacher to personalize their 
instruction). The schools will be using the individual rotation model, which allows students 
within a given course or subject to rotate on an individually customized, fixed schedule 
between online courses and a variety of classroom environments with face-to-face teach- 
ers. Individual students have their own netbook and a unique schedule. The model also 
includes site coordinators and a principal who is involved in daily classroom observation. 
The goal of the model is to improve student achievement and individualize instruction. 
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Review of logic models 


Supporting materials for this section: slides 49-52. 

Here are a few quick reminders about what a logic model is and what it is not. 

A logic model is: 

• A graphic representation of the theory of change driving a program or policy. 

• A framework for planning, implementation, monitoring, and evaluation. 

A logic model is not: 

• A strategic or fully developed plan for designing or managing a program or policy. 

• An evaluation design or evaluation method. 

While a logic model is not a strategic or fully developed plan or an evaluation design or 
evaluation method, it can be useful in developing any of these more detailed resources. 
The focus of session II of the workshop is on the latter: How does a logic model support the 
development of an evaluation plan for a program or policy? 

If participants have created draft logic models for a program or policy they are engaged in 
or considering in their work, the logic model drafts will serve as the template to guide their 
work throughout this session. If not, they may use the College Ready case logic model on 
slide 51 (or see the two example logic models, one for the College Ready example and one 
for an educator evaluation system, in appendixes C and D). 

Consider the following questions when evaluating a draft logic model: 

• What elements of the logic model were hardest to develop? 

• Is the problem statement the right “grain size”? 

• Within the strategies and activities, did you identify overarching strategies? 

• What assumptions did you uncover? 

• What is the timeframe for your outcomes? 

• What are the impacts? 

• What was your process for developing the model? 

• What requires further explanation or discussion? 
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Introducing evaluation 


Supporting materials for this section: slides 53-57. 

Program and policy evaluation helps answer important questions that inform this work. At 
a basic level, evaluation answers the questions: Are we successful? Have we had an impact? 
What are the most influential aspects of the program? 

More specifically, evaluations ask questions such as: 

• Is the program or policy effective? 

• Is the program or policy working as intended? 

• What aspects of the program are working? What aspects are not working? 

High-quality evaluation is designed to support your work, inform what you do, and 
enhance your impact. To do so, evaluation should be considered at the onset of program 
and policy design, ideally when the logic model is being developed. In other words, as a 
program or policy is conceived, evaluating the same program or policy should be a part of 
the conversation, by asking questions such as: 

• What do we anticipate to be the impact of this policy? 

• How will we know if we are successful? 

• What do we think will be the most influential aspects of the program? 

All these questions suggest directions for evaluation. Do not wait until the program or 
policy is in the midst of implementation to begin to consider these questions and how to 
answer them. Invest early in considering these questions and designing an evaluation that 
will help answer them. It may also be helpful to involve others, including staff and partici- 
pants, in helping plan the evaluation. 


Activity II. 1: How will you know? 

Consider your own program or policy logic model. How will you know if one or more of your strat- 
egies have been successful? Brainstorm some ways you will know your efforts have yielded the 
results you hope to achieve. 
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Evaluations generally have one of two purposes: 

• Improve: These are formative, process, or implementation evaluations. 

• Prove: These are summative, results, or outcome evaluations. 

This workbook refers to these evaluation purposes as formative (improve) and summative 
(prove). Most evaluation questions emerge from the strategies and outcomes sections of the 
logic models. You want to know about the strategies that you are trying and how they are 
going, and you want to know about outcomes and impact. 

Generally, evaluations that focus on strategies (and outputs) are formative or process evalu- 
ations, or evaluations that are designed to help guide changes or improvements. Evaluations 
that focus on the outcomes in the logic model are generally summative evaluations — 
designed to prove the value, merit, or impact of the program or policy. 

There are generally four types of evaluations: 

• Needs assessment (formative). This type of evaluation determines what is needed 
(at the onset) and helps set priorities (for example, is more money needed to 
support blended learning?). These types of evaluations are often designed to help 
create or build a program or policy, so a logic model might be developed after the 
needs assessment. In fact, the needs assessment might provide information that 
helps clarify the problem to which the program or policy is designed to respond. 

• Process evaluation (formative). This type of evaluation examines what goes on 
while a program is in progress. The evaluation assesses what the program is, how it 
is working, whom it is reaching, and how (for example, are participants attending 
as anticipated?). 

• Outcome evaluation (summative). This type of evaluation determines the results 
from and consequences of a program, generally for the people most directly affect- 
ed by the program (for example, did participants increase their knowledge or 
change their attitudes or behavior?). 

• Impact evaluation (summative). This type of evaluation determines the net causal 
effects of the program beyond its immediate results. Impact evaluation often 
involves comparing what appeared after the program with what would have 
appeared without the program. These evaluations generally include comparison 
groups, interrupted time series, or other designs that allow evaluators to capture 
what happened to the target compared with what would have happened without 
the program (for example, achievement scores and acceptance rates). 
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Moving from logic model to evaluation questions 


Supporting materials for this section: slides 58-63. 

The purpose of this section is to make the connection between the logic model and devel- 
opment of appropriate evaluation questions, using the logic model as a basis for developing 
the questions. The first step in making the transition from the logic model to a potential 
evaluation is to consider the questions that are derived from the model that you may want 
answered. 

Developing evaluation questions 

As noted in the previous section, some questions ask about improvements to the program 
or policy (formative, process, and implementation questions), while others ask about the 
impacts (summative, results, and outcome questions). Generally: 

• Formative questions (improve) are asked while the program is operating and are 
for the purpose of program improvement or midcourse correction. 

• Summative questions (prove) are asked at completion of or after the program and 
are for the purpose of determining results and assessing effectiveness. 

Regardless of the type of questions, there are some guiding questions to consider for all 
evaluation questions. 

• Can the questions be answered given the program? One of the main reasons for 
building a logic model as part of program evaluation is to determine what ques- 
tions are appropriate based on the program. By describing what the program is, 
the logic model helps determine what is appropriate to evaluate. 

• Are the questions high-priority? Try to distinguish between what you need to 
know and what might merely be nice to know. What are the key, most important 
questions? For whom? Why? 

• Are the questions practical and appropriate to the capacity you have to answer 
them? Consider time, resources, and the availability of assistance needed to answer 
the questions. As appropriate, bring stakeholders together and negotiate a practi- 
cal set of questions. Remember, it is better to answer a few questions thoroughly 
and well. 

• Are the questions clear and jargon-free? Apply the “Great Aunt Lucy test.” Would 
someone like your Great Aunt Lucy or anyone who is not steeped in the language 
of your particular field understand the question? Avoid the use of jargon or vague 
words that can have multiple meanings. Always define key terms so that everyone 
understands the meaning. 
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Activity 11.2: Formative and summative evaluation 


Come up with a formative and summative evaluation question for one of the sample cases, 
such as the blended learning case, or for a program or policy from your own work. 

• Formative evaluation: 

Topic: Blended learning or 

Question: 


• Summative evaluation 
Topic: Blended learning or 
Question: 


Considering the audience 

Another key aspect of developing good evaluation questions is considering different 
audiences, or the different stakeholders for a program and policy, the different types of 
questions they might have, and how they would use the answers to these questions (for 
example, what decisions would result from answers). 

Table 1 outlines some traditional audiences, the types of questions they are likely to have, 
and how they might apply answers to these questions to make decisions. 
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Table 1. Traditional audiences, evaluation questions, and uses 

1 Audience 

Typical questions 

Evaluation use | 

Program staff 

Are we reaching our target 
population (for example, high 
school students; low-income 
families with preschool-age 
children)? 

Are participants in the program 
engaged? Satisfied? 

Is the program being run well? 

Flow can we improve the program? 

Day-to-day program operations; 
changes in program design and 
delivery 

Participants 

Is the program helping people like 
me? 

Flow could the program better 
serve my needs? 

Flow could 1 get more out of the 
program? 

Decisions about participation or 
value to them 

Public officials 

Who does the program serve? 

Is it reaching the target 
population? 

What difference does the program 
make? 

Are participants engaged and 
satisfied with the program? 

Is the program cost-effective? 

Decisions about support, 
commitment, funding, scale-up, 
and duplication 

Funders 

Is the program meeting its goals? 
Is the program worth the cost? 

Decisions about ongoing funding; 
accountability 

Source: W.K. Kellogg Foundation, 2006. 
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Activity 11.3: Generating questions for different audiences 

Think about your own context and consider: 

• Audience: Who are the different members of each stakeholder group (staff, participants, 
and the like)? 

• Questions: What questions might different stakeholders have about the program or policy? 

• Evaluation use: How might these different stakeholders use the answers to these 
questions? 


Audience Questions Evaluation use 


Program staff 


Participants 


Public officials 


Funders 
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Generating indicators 


Supporting materials for this section: slides 65-67. 


Activity 11.4: Generating indicators of the flu 

How do we know a child has the flu? 

Take a moment to brainstorm how we know a child has the flu: 


Perhaps we feel her forehead for fever, listen to her sniffles, notice her lethargy, recognize 
a drop in appetite, and eventually take her temperature. All of these are indicators of the 
flu. They do not mean absolutely that the child has the flu, but they do provide specific and 
measurable evidence that suggest the flu. 

In developing evaluation questions, you must consider how you will know that you have 
achieved the program or policy goals and therefore answered the questions of interest. 
This section demonstrates how the logic model can support generation of good indicators 
of program or policy success. 

If asking “Is the program successful?”, some understanding is needed of how “success” is 
measured. In other words, the question, “How will we know the program is successful?” has 
to be answered. The logic model provides some support for this. 

Starting with the logic model’s outputs and outcomes, indicators can be developed that 
answer the question, “How will we know the program is successful?” Indicators are differ- 
ent from the outputs or outcomes included in the logic model: While the outputs or the 
outcomes are more general goals for program implementation or outcomes, indicators are 
specific, measurable targets related to the outcomes of interest. 

In short, indicators are: 

• Specific, measurable targets. 

• Seen, heard, read, and felt. 

• Connected to strategies, activities, outputs, and outcomes. 

• Evidence representing phenomenon of interest (such as the outcome). 

For example, if the outcome is increased parent engagement, the indicator is a specific per- 
centage of parents engaged or a specific increase in number of parents engaged. It is these 
measurable indicators that lead eventually to an answer to the question, “Is the program 
successful?” 
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Indicators do not absolutely mean that a policy or program is responsible for the results we 
measure. To use an example from the College Ready case, a parent engagement program 
(the program) may not be responsible for the rise in college applications among student 
participants (the indicator). There might be other factors, such as a decline in college costs 
or a particularly influential teacher at the school encouraging and supporting applications, 
that lead to an increase in the number of college applications submitted — rather than the 
increase being a direct result of the program. But this increase in college applications (the 
indicator) could reasonably be attributed to a program that works with the students and 
their parents to support college readiness. 

Using the logic model to generate indicators 

Supporting materials for this section: slides 68-76. 

Just as the logic model follows the basic format from inputs (resources) to outputs (strate- 
gies or activities) to outcomes (from short term to long term to impact), this same logic is 
used to generate indicators. 

As stated above, indicators are related to the logic model categories of resources, strategies 
or activities, and outcomes or impact. They go a step further and provide clear numbers 
or percentages, when appropriate, associated with these resources, activities, outputs, or 
outcomes — or measurable evidence of the phenomenon of interest represented by the 
outputs or outcomes. 

• Indicators related to inputs provide information about the resources used, the 
timeliness of the resources, and the relevance of the resources (whether tangible 
or intangible). Indicators related to these inputs may help answer questions about 
impediments or facilitators of implementation. 

• Indicators related to outputs capture the numbers or percentages of workshops 
presented, the numbers of participants, and other data that provide information 
about whether the program was implemented as intended. Did it do what it set out 
to do? Did it reach the right people? 

• Indicators related to outcomes or impacts provide data about the results of partic- 
ipation, such as changes in knowledge, skill, behavior, and attitudes among indi- 
viduals or groups targeted by the program or policy. 

For example, if the strategy or activity was to deliver a parent education class, an indicator 
related to that activity might be the number of classes delivered or the number of parents 
who attended (indicators related to the activities and outputs, or process indicators). If an 
outcome is increased parent understanding of the college application process, an indicator 
would be the number or percentage of parents reporting increased understanding (an indi- 
cator related to the outcomes, or an outcome indicator). 

When generating indicators based on the various elements of the logic model (inputs, 
outputs, strategies and activities, and outcomes), ask these basic questions: 

• What would achieving the goal reflected in the outcome look like? 

• How would we know if we achieved it? 

• If I were visiting the program, what would I see, hear, or read that would tell me 
that the program is doing what it intends? 
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Activity 11.5: Process and outcome indicators 

Using the College Ready program case, this table is designed to help you map a path from an 
activity in the logic model to an output to an outcome to an indicator. 


Activity 

Deliver a set of parent workshops for college readiness. 

Output 

Six workshops developed and delivered; 100 parents recruited to participate. 

Outcome 

Parents increase their understanding of college application process. 

Indicator 

Process: 

Outcome: 


Identifying the right indicators 

Supporting materials for this section: slides 77-83. 

Different indicators are related to different types of questions. 

• To know whether the program has sufficient resources or funding to operate, look 
at indicators related to the program inputs. 

• To know whether the program was implemented as intended, look at indicators 
related to the strategies and activities or outputs. 

• To learn the ultimate value and impact of the program, look at outcome-related 
indicators. 

Some indicators may be more straightforward or easier to measure than others. Sometimes 
one indicator is all that is needed for a clear explanation. For example, school graduation 
rate might be the agreed on indicator for the outcome of decreasing the school dropout 
rate. In other cases, more than one indicator is necessary to capture a more complex 
outcome. 
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If the outcome of interest is improved parent involvement in school, several indicators may 
be necessary, such as: 

• Attendance at school meetings. 

• Participation in parent-school organization. 

• Parent calls made to the school. 

• Attendance at school functions. 

Quantitative and qualitative indicators 

Supporting materials for this section: slides 84-87. 

Indicators may be quantitative or qualitative. Given the current interest in and demand 
for measurable outcomes, evaluation questions often focus only on the outcomes in a logic 
model and only on the quantitative measures associated with those outcomes. However, 
to attribute quantitative outcomes (such as graduation rates or improvements on standard- 
ized tests) to a program, it is important to ask questions about the process that may have 
contributed to those outcomes. There are some questions that are best answered with a 
mixture of both quantitative and qualitative data. This suggests the need for both qualita- 
tive and quantitative indicators to answer questions of interest. 

• Quantitative data are generally best suited to summative evaluations — such as 
information related to proving the value or impact of the program or policy. 

• Qualitative data are generally better suited to formative evaluations — such as 
those that focus on how to improve the program. 

However, this is not to suggest that all formative evaluations should be qualitative and 
that all summative evaluations should be quantitative. Often, a mix of measures is the best 
approach. Qualitative data, collected through interviews, observations, and other methods, 
often provide the depth of information needed to interpret quantitative data, such as test 
scores or graduation rates. We often want to know both whether a program or policy had 
the desired impact and how. Thus, a mix of measures is advisable. 

It is a good idea to collect different types of data (for example, quantitative and qualitative) 
from several sources (for example, surveys, interviews, grades) and from different groups 
of stakeholders (for example, students, parents, mentors, staff, partners such as schools). 
While some funders may prefer quantitative data on outcomes, others, such as staff or 
parents, may prefer qualitative data from parents or students. 

For example, in the College Ready case, to learn whether the program increased student 
interest in college, the indicators might include both quantitative (number of applications 
completed) and qualitative data (guidance counselors’ reports on student interest). 

A few final considerations about indicators. Indicators may: 

• Match the outcomes of interest or questions asked. 

• Be singular for a given outcome or question. 

• Be quantitative or qualitative. 

• Vary based on the audience. 
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Building an evaluation design 


Supporting materials for this section: slides 88-91. 

This section provides tools for building an appropriate evaluation design. 

Once you have generated a good logic model, come up with the best evaluation questions, 
and developed what you believe will be the best indicators of program or policy success 
(either for program improvement or to make the case for program impact), you are ready to 
build the evaluation design. 

Determining what type of evaluation is best for you 

The first question to answer about evaluation design is quite basic: Is the purpose of the 
evaluation to examine process elements (formative) or to examine overall impact (summa- 
tive), or is it a hybrid evaluation, with some of each? Answering this question should help 
clarify what to include in the evaluation. 

You should also return to the question of audience. Who is the audience for the evalua- 
tion, what does the audience want to know, and how will the information be used? 

Consider your capacity: Who will conduct the evaluation, using what resources, and 
within what timeframe? 

Finally, assuming there are some financial, time, or other capacity constraints, what are 
your priorities? What do you need to know? 

Identifying appropriate data sources 

When choosing measures for program evaluation, think about the data collection needs as 
well as data collection capacity. 

• Access pre-existing data. Consider collecting from pre-existing sources of data, such 
as school attendance records or items from surveys that a school district already 
requires students to take, that will meet your evaluation needs. 

• Use existing instruments. When data cannot be collected through existing sources, 
consider existing instruments that measure the same concepts that you are looking 
to measure. These instruments may be effective as written, a small tweak may be 
needed, or you may need to adopt only a few items from the entire instrument. It 
may even be possible to add these instruments or items to existing surveys current- 
ly being administered in your school or district. 

Consider the types of data sources that might serve as indicators of success, both for pro- 
cess-related questions and for outcome-related questions. 

The following are data sources that may be useful for collecting process and outcome data: 

• Administrative data (program documents, activity logs, registration records, and 
the like). 

• Focus groups. 

• Interviews. 
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• Observations. 

• Surveys. 

• Student test scores and grades. 

• Teacher assessments. 


Activity 11.6: Consider data sources 

Consider the data sources available to you in your program. Brainstorm what relevant data 
sources you already collect, either because they are required or because you choose to collect 
them: 


Creating a data collection framework 

Supporting materials for this section: slides 92-94. 

In addition to identifying potential data sources, the details of data collection must be 
ironed out early in planning for the evaluation. It is important to develop a clear sense of 
what types of data are available, who collects and manages the data, and when the data are 
collected. Developing a data collection framework linked to the activities and outcomes of 
interest will help guide the evaluation process. In some cases, the data collection framework 
may look different for process evaluation and outcome evaluation. However, in many cases 
an evaluation will include some process, or formative, and some outcome, or summative, 
components. As such, these may be combined into one framework, as in the table below. 

Case: College Ready program 


Strategy 
or activity 

Output or 
outcome 

Formative 

Summative 

Indicator 

Data source 

Data 

collection 

instrument 

When 

collected 

By whom 

Parent 

education 

strategy 

High rate of 
parent at- 
tendance at 
workshops 

✓ 


70 percent 
of parents 
attend five 
out of six 
workshops 

Administrative 

data 

Attendance 
log at work- 
shops 

At beginning 
of each 
session 

Program 

director 

Parent 

education 

strategy 

Increased 
parent under- 
standing 
of college 
application 
process 


✓ 

85 percent of 
parents who 
attend more 
than four 
workshops re- 
port increased 
understanding 

Parent feed- 
back 

Survey and 
interviews 

Beginning 
of program, 
end of pro- 
gram 

Program 

staff 
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It may be useful to distinguish among short-term, long-term, and impact data when creat- 
ing an outcome-specific data collection table (see the table below). The reason for doing 
this may be relevant to only some evaluations, depending on the depth and duration of the 
evaluation plan. However, distinguishing among outcomes in the logic model should help 
guide the types of data that ought to be collected (and when) to reflect those outcomes. 
For example, if changes in student test scores are not anticipated until a program has been 
up and running for three to five years, these data should not be collected (except to serve 
as a baseline, or point from which change will be measured) until the long-term phase of 
data collection. 


Strategy or 
activity 

Output or 
outcome 

Indicator 

Data source 

Data collection 
instrument 

When collected 

By whom 

Short term 

Parent education 
strategy 

Increased parent 
understanding of 
college applica- 
tion process 

85 percent of 
parents who 
attend more than 
four workshops 
report increased 
understanding 

Parent feedback 

Survey and inter- 
views 

Beginning of 
program, end of 
program 

Program staff 

Long term 

Student education 
strategy 

Increased student 
understanding of 
college applica- 
tion process 

80 percent of stu- 
dents who attend 
the workshops ap- 
ply to college and 
get accepted to at 
least one college 
or university 

Student feedback 

Survey and inter- 
views 

End of program 

Program staff 


Impact 
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Additional activity: Data collection framework 


Try it yourself. Consider your program or policy and generate a data collection strategy. This activity is best complet- 
ed after having developed a full logic model. 


Strategy or 
activity 

Output or 
outcome 

Indicator 

Data source 

Data collection 

instrument When collected By whom 

Short term 


Long term 


Impact 


44 






Putting it all together 


Supporting materials for this section: slides 95-97. 

Now that you have a logic model, a set of measurable indicators, some evaluation ques- 
tions, a data collection framework, and at least some idea about evaluation design, you are 
nearly ready to proceed with evaluation. What you still need is the overall evaluation plan 
and timeline. This section provides some tools to help you develop your plan and timeline. 

At this point, you have learned about all the steps necessary to get you to the evaluation. 
In brief, you have learned about how to: 

• Develop a logic model, in collaboration with stakeholders. 

• Clarify who the audience is for the evaluation and how it will be used. 

• Identify and prioritize evaluation questions, based on the logic model. 

• Select appropriate indicators, based on the outcomes of interest. 

• Identify data sources and a data collection plan (including considering whether 
the evaluation is serving a formative/process or a summative/outcome goal). 

• Consider evaluation design, with awareness of resources, capacity, and timeline. 

Once you have completed all these steps, it is still a good idea to develop an evaluation 
prospectus that lays out much of this information in clear, narrative form. 

Evaluation prospectus 

If you are going to look for an external evaluator, or even if you are going to do the eval- 
uation in house, it is wise to have an evaluation prospectus to accompany all the other 
materials you are generating. This prospectus should provide a clear and straightforward 
answer to the following questions: 

• What are you going to evaluate? 

• What is the purpose of the evaluation? 

• How will the results of the evaluation be used? 

• What specific questions will the evaluation answer? 

• What data sources will be necessary to answer these questions? 

• How will the data be analyzed (evaluation design)? 

• What resources are needed to conduct this evaluation? 

• What is the timeline for the evaluation? 

• How will the results be shared or disseminated? 

• Who will manage the evaluation? 
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Evaluation timeline: Example of Gantt chart 


One more tool may be helpful in implementing the evaluation plan. Evaluators often work 
with Gantt charts, which are a kind of timeline, displayed in such a way that readers can 
immediately see a proportionate, chronological account of the time for each evaluation 
task. They can be helpful for clarifying the sequence of activities in an evaluation, moni- 
toring progress toward goals, and communicating with others about the products associat- 
ed with the evaluation. 

• The vertical axis shows the tasks to be completed. 

• The horizontal axis shows the time scale. 

• A shaded area indicates how long each task will take. 

• Milestones are denoted with a ★. 


Task January February March April May June July 


Develop survey 


Select sample 


Administer survey 


Analyze survey data 


Compare survey data 
to indicators 


Write up findings 


☆ 
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Review 


Supporting materials for this section: slides 98-99. 

When introducing logic models, session I noted that they are a useful tool for program 
design, implementation, and evaluation. They should be living documents that are referred 
to throughout the life of the program and the evaluation and amended as needed. They 
are also helpful for guiding a program as it evolves and for ensuring that the work of the 
program remains focused on the key goals and outcomes. 

Logic models are useful for program evaluation, especially when evaluation is designed 
in concert with the logic model. It is much better to consider evaluation at the outset of 
a program’s or policy’s development rather than as an afterthought or halfway through 
program implementation. 

Finally, engaging key voices — staff, parents, students, funders, and others — in discussions 
about program design and evaluation will promote the buy-in and ongoing support of these 
participants as well as increase the authenticity of the model and the evaluation. 
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Appendix A. Simple logic model 


Problem statement 


Resources 


Strategies 


Outputs 


Short-term 


Long-term 

(inputs) 


and activities 



outcomes 


outcomes 


Impacts 


Assumptions 


A-l 










Appendix B. Template for simple logic model 


Problem statement: 


Resources 

Strategies and activities 

Outputs 

Short-term outcomes 

Long term outcomes 

Impacts 

What resources are or could 
reasonably be available? 

What will the activities, 
events, and such be? 

What are the initial products 
of these activities? 

What changes are expected 
in the short term? 

What changes do you want 
to occur after the initial 
outcomes? 

What are the anticipated 
changes over the long haul? 


CD 

p 


Assumptions: 




Appendix C. Sample logic model for college readiness program 


Problem statement: Low-income high school students in selected communities attend college at a lower rate than their middle-class peers, leading to more limited opportunities, higher rates 
of unemployment, and lower earnings. 


Resources 

Strategies and activities 

Outputs 

Short-term outcomes 

Long term outcomes 

Impacts 

What resources are or could 
reasonably be available? 

What will the activities, 
events, and such be? 

What are the initial products 
of these activities? 

What changes are expected 
in the short term? 

What changes do you want 
to occur after the initial 
outcomes? 

What are the anticipated 
changes over the long haul? 

Partnership with three 

• Establish local college 

• Recruited adequate num- 

Participating students ap- 

• Participating students are 

■ Low-income students in 

public high schools. 

mentorship program. 

ber of mentors for student 

ply to at least one college 

accepted to and attend 

participating communi- 

■ Community mentors. 

Recruit and prepare peer 

cohort. 

on time. 

college, remaining enrolled 

ties attend college at the 

• Local university space for 

mentors. 

• Developed and delivered 

Parents report increased 

into the third semester of 

same rate as middle-class 

parent meetings. 

■ Develop and deliver 

12 workshops on college 

understanding of the 

college. 

peers. 

■ Volunteer college 

student readiness program 

application process, SAT/ 

college application 

• Participating students’ 

• Low-income students in 

admissions directors for 

(workshops). 

ACT, FAFSA, and college 

process. 

GPAs are above 3.0 at 

participating communities 

application workshop. 

■ Develop and deliver parent 

life. 

• Students report increased 

college, into the third 

graduate from college at 

• Student volunteers 
for childcare at parent 
meetings. 

education (workshops). 

• Developed and delivered 
six workshops for parents. 

• High interest and 
attendance demonstrated 
at all workshops for 
parents and students. 

readiness for college. 
Participating students 
complete FAFSA forms on 
time. 

semester. 

■ Increased parental 
engagement in 
participating high schools’ 
student education. 

the same rate as middle- 
class peers. 

Participating high schools 
see increase in parent and 
student engagement. 
Participating high schools 
state test scores increase 
by x percent. 


Assumptions: College attendance is desired goal for participating communities; high school leaders will remain consistent and support program; parents will show interest and participate in 
program. 




Appendix D. Example of a logic model for an 
educator evaluation system theory of action 


Context: Implement four pillars of effective teaching: teacher evaluation, teacher preparation, teacher induction, and 
professional development. 


Long-term goals: To support teachers’ continuous improvement and to improve student learning and other outcomes. 


Evaluation goal: To evaluate the implementation of the state teacher evaluation framework in a subset of schools and districts in 2012/13. 


Strategies 

Participants 

Implementation 

outcomes 

Intermediate 

outcomes 

Impact and long-term 
outcomes 

Design and imple- 

Teachers 

District teacher 

Based on infor- 

Alignment of evalua- 

ment new teacher 

Classroom and school 

evaluation plans 

mation gained 

tion and professional 


evaluation systems 
that employ multiple 
measures including: 
Teacher 
observations. 
Evaluator 
and teacher 
conferences. 

• Student-learning 
objectives. 

• Professional 
growth plans that 
place teachers in 
"tracks" based on 
experience. 

Multiple rating 
categories. 
Teaching standards 
based on the Dan- 
ielson framework. 3 


specialists 

School and district 
administrators 

Department of 
Education 

Students 

Parents 

Other stakeholders 
(such as school 
committees, school 
board) 





correspond with state 
guidelines. 

District teacher eval- 
uation systems have 
been implemented as 
designed with regard 
to multiple measures 
of student learning 
and teacher perfor- 
mance including: 
Teacher 
observations. 
Evaluator 
and teacher 
conferences. 

■ Student-learning 
objectives. 

■ Professional growth 
plans. 


from evaluation of 
implementation: 
Program changes 
and modifications 
to framework for 
teacher evaluation. 

• Department of 
Education provides 
guidance, training, 
and support materi- 
als for districts. 

• Individual district 
modifications to 
teacher evaluation 
systems. 

• Systems differen- 
tiate among teach- 
ers based on level 
of experience and 
performance. 


support systems 

Improvement in teach- 
er practice. 

Improvement in 
student-learning 
outcomes, including: 
Higher achievement 
on standardized 
assessments. 
Decreased dropout 
rate. 


District teacher 
evaluation systems 
have been implement- 
ed with fidelity with 
regard to use of multi- 
ple rating categories. 


a. Danielson, C. (2007). Enhancing professional practice: A framework for teaching. Alexandria, VA: Association for Supervision and 
Curriculum Development. 


Assumptions: The Department of Education assumes that educator evaluation is one of four essential pillars of effective teaching. To 
maximize long-term outcomes, all pillars need to be in place and integrated. The department also assumes that if districts implement 
reformed teacher evaluation systems that map onto the Blueprint for Effective Teaching in New Hampshire (http://www.education. 
nh.gov/teaching/documents/phaselreport.pdf), then district-level systems will lead to long-term outcomes indicated above. It 
hypothesizes that achievement of outcomes will be influenced by professional climate: Schools with a favorable climate will be more 
likely to have successful implementation. 


D-l 
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